Why testers?

My sister got her kids a little puppy, and they’ve been trying to train it. To live with a dog in the house, you need to teach it not to jump on people, not to poop in the house, to sit on command, and to never, ever, ever chew on the iPad. Never. Good girl.

With dogs the main trick to training is that feedback has to be immediate. If you come home to discover that, hours before, the dog tipped over the garbage can in the kitchen, it’s too late for training. You can yell at her but she just won’t get what you’re going on about. Dogs are just not that smart.

For programmers, getting better at what you do requires quick feedback, positive and negative, on what you’ve just done. The faster you get the feedback, the faster you’ll learn. With long-cycle shrinkwrap software, it can take a year or more to hear feedback from customers.

That’s one of the reasons we have testers. A great tester gives programmers immediate feedback on what they did right and what they did wrong. Believe it or not, one of the most valuable features of a tester is providing positive reinforcement. There is no better way to improve a programmer’s morale, happiness, and subjective sense of well-being than a La Marzocco Linea espresso machine to have dedicated testers who get frequent releases from the developers, try them out, and give negative and positive feedback. Otherwise it’s depressing to be a programmer. Here I am, typing away, writing all this awesome code, and nobody cares. Boo hoo.

Who should be a tester? That’s tricky! Software testing is one of those careers that isn’t that well known, so a lot of people who would be great at testing and would probably enjoy it a lot never consider applying for jobs as testers.

Signs of a good tester:

  • Scientific
  • Loves a good puzzle, even the kind that takes days to solve
  • Likes to think about things methodically
  • Generally likes working with software and computers

You don’t have to be a programmer to be a tester. A lot of companies want testers to be programmers who write automated test suites. It seems more efficient that way. This reflects a misunderstanding of what testers are supposed to do, which is evaluate new code, find the good things, find the bad things, and give positive and negative reinforcement to the developers. Sure, automated test suites are a time saver, but testing software covers so much more than that. If you put too much emphasis on those scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.

A particularly terrible idea is to offer testing jobs to the programmers who apply for jobs at your company and aren’t good enough to be programmers. Testers don’t have to be programmers, but if you spend long enough acting like a tester is just an incompetent programmer, eventually you’re building a team of incompetent programmers, not a team of competent testers. Since testing can be taught on the job, but general intelligence can’t, you really need very smart people as testers, even if they don’t have relevant experience. Many of the best testers I’ve worked with didn’t even realize they wanted to be testers until someone offered them the job.

If you:

  • Love software and computers
  • Want to work on a software team, and
  • Don’t particularly like programming

you should consider being a tester. (We’re hiring! What a coincidence!)

Figuring out what your company is all about

What is your company about?

Recently I got inspired by Kathy Sierra, whose blog Creating Passionate Users and Head First series of books revolutionized developer education. She kept saying the same thing again and again: help your users be awesome.

Kathy taught me that if you can’t explain your mission in the form, “We help $TYPE_OF_PERSON be awesome at $THING,” you are not going to have passionate users. What’s your tagline? Can you fit it into that template?

It took us nine years, but we finally worked out what Fog Creek Software is all about, which I’ll tell you in a moment, but first, some backstory.

In the early days, we were all about making a great place to be a software developer in New York City.

Yep, that was all there was to it. Almost every software job in the city was terrible. You had a choice of which kind of terrible. Want to wear a suit and work long hours under crummy conditions? Take a job at a bank. Want to report to a manic-depressive creative who demands that you stretch HTML in ways that would have you put to death, in certain countries? Take a job at a media company. Want to work 24/7 in a basement with water pipes dripping on your head and get paid in worthless stock options? Take your pick of the revenue-free dotcom startups.

Why New York, then? There are lots of great product companies where software developers are treated very well in Redmond, Washington. But I was sick of trying to live in lesser cities. Sure, the Seattle area is beautiful, and green, and clean, and possesses great coffee, and I understand that there are even a couple of grocery stores open late now. But I’m staying in New York, because it’s the greatest city in the world.

I gave up the search, and decided to start a company with my buddy Michael Pryor. Making a nice place to work was our primary objective. We had private offices, flew first class, worked 40 hour weeks, and bought people lunch, Aeron chairs, and top of the line computers. We shared our ingenious formula with the world:

The tagline was “building the company where the best software developers want to work.” It was, to say the least, awkward. It didn’t make for a good elevator pitch. It didn’t really have the right format. “Abercrombie and Fitch: building the apparel store where the hottest teenagers will want to work.” Who cares? Not the hot teenagers, I’ll tell you that.

Anyway we accomplished that goal. Cross it off the list. What’s next? We needed a new mission statement.

And it has to be something of the form, “We help $TYPE_OF_PERSON be awesome at $THING.”

Bells went off. Everything we’ve done successfully has one thing in common: It’s all about helping software developers be awesome at making software.

That includes Joel on Software, Stack Overflow, all the books I’ve been writing, the conferences like DevDays and Business of Software, the Jobs Board and Stack Overflow Careers.

It includes our flagship product, FogBugz, which is all about giving developers tools that gently guide them from good to great. It’s the software implementation of the philosophy I’ve been writing about for a decade, lacking only one thing: the feature to replace exceptions with return values, while adding Hungarian prefixes to all variable names. THAT IS A JOKE, PEEPLE. Put DOWN the bazooka.

Helping you make more awesome software is why I write endlessly about what we’re doing at Fog Creek, despite the fact that people accuse me of shilling. I’m not writing to promote our products. You don’t have to buy our products to get the benefit of reading about my experience designing them and building them and selling them. I’m writing to share some of my experiences in case they can help you make better software.

Our focus on helping developers explains why one of our early products, CityDesk, flopped: it had nothing to do with software developers. And it explains why another of our products, Fog Creek Copilot, only found a market in the niche of software developers doing tech support.

So, here you go, the new tagline: “We help the world’s best developers make better software.”

Going through this exercise made it easy to figure out what belongs in future versions of FogBugz and what doesn’t. In particular, we’re adding source control and code review features to FogBugz, using Mercurial, the best open-source distributed version control system. Everything that helps developers make better software belongs in FogBugz: project planning, project management, bug tracking, and customer service.

It took almost ten years, but I think we finally got the mission for the next ten nailed.

Optional Advertainment: If you’ve got a moment, check out this 4½ minute trailer for Make Better Software, a new video training series we’ve been working on for more than a year. It’s the video edition of Joel on Software and fits perfectly with our agenda of helping developers make great software.

Martian Headsets

You’re about to see the mother of all flamewars on internet groups where web developers hang out. It’ll make the Battle of Stalingrad look like that time your sister-in-law stormed out of afternoon tea at your grandmother’s and wrapped the Mustang around a tree.

This upcoming battle will be presided over by Dean Hachamovitch, the Microsoft veteran currently running the team that’s going to bring you the next version of Internet Explorer, 8.0. The IE 8 team is in the process of making a decision that lies perfectly, exactly, precisely on the fault line smack in the middle of two different ways of looking at the world. It’s the difference between conservatives and liberals, it’s the difference between “idealists” and “realists,” it’s a huge global jihad dividing members of the same family, engineers against computer scientists, and Lexuses vs. olive trees.

And there’s no solution. But it will be really, really entertaining to watch, because 99% of the participants in the flame wars are not going to understand what they’re talking about. It’s not just entertainment: it’s required reading for every developer who needs to design interoperable systems.

The flame war will revolve around the issue of something called “web standards.” I’ll let Dean introduce the problem:

All browsers have a “Standards” mode, call it “Standards mode,” and use it to offer a browser’s best implementation of web standards. Each version of each browser has its own Standards mode, because each version of each browser improves on its web standards support. There’s Safari 3’s Standards mode, Firefox 2’s Standards mode, IE6’s Standards mode, and IE7’s Standards mode, and they’re all different. We want to make IE8’s Standards mode much, much better than IE7’s Standards mode.

And the whole problem hinges on the little tiny decision of what IE8 should do when it encounters a page that claims to support “standards”, but has probably only been tested against IE7.

What the hell is a standard?

Don’t they have standards in all kinds of engineering endeavors? (Yes.)

Don’t they usually work? (Mmmm…..)

Why are “web standards” so frigging messed up? (It’s not just Microsoft’s fault. It’s your fault too. And Jon Postel’s (1943-1998). I’ll explain that later.)

There is no solution. Each solution is terribly wrong. Eric Bangeman at ars technica writes, “The IE team has to walk a fine line between tight support for W3C standards and making sure sites coded for earlier versions of IE still display correctly.” This is incorrect. It’s not a fine line. It’s a line of negative width. There is no place to walk. They are damned if they do and damned if they don’t.

That’s why I can’t take sides on this issue and I’m not going to. But every working software developer should understand, at least, how standards work, how standards should work, how we got into this mess, so I want to try to explain a little bit about the problem here, and you’ll see that it’s the same reason Microsoft Vista is selling so poorly, and it’s the same issue I wrote about when I referred to the Raymond Chen camp (pragmatists) at Microsoft vs. the MSDN camp (idealists), the MSDN camp having won, and now nobody can figure out where their favorite menu commands went in Microsoft Office 2007, and nobody wants Vista, and it’s all the same debate: whether you are an Idealist (”red”) or a Pragmatist (”blue”).

Let me start at the beginning. Let’s start by thinking about how to get things to work together.

What kinds of things? Anything, really. A pencil and a pencil sharpener. A telephone and a telephone system. An HTML page and a web browser. A Windows GUI application and the Windows operating system. Facebook and a Facebook Application. Stereo headphones and stereos.

At the point of contact between those two items, there are all kinds of things that have to be agreed, or they won’t work together.

I’ll work through a simple example.

Imagine that you went to Mars, where you discovered that the beings who live there don’t have the portable music player. They’re still using boom boxes.

You realize this is a huge business opportunity and start selling portable MP3 players (except on Mars they’re called Qxyzrhjjjjukltks) and compatible headphones. To connect the MP3 player to the headphones, you invent a neat kind of metal jack that looks like this:

Picture of a stereo headphone jack

Because you control the player and the headphone, you can ensure that your player works with your headphones. This is a ONE TO ONE market. One player, one headphone.

[1] to [1] 

Maybe you write up a spec, hoping that third parties will make different color headphones, since Marslings are very particular about the color of things that they stick in their earlings.

...may not touch the connecting block f, an insulating washer g, is placed under the screw and washer h. The insulating washer is made large enough so that there will be no stray strands...

And you forgot, when you wrote the spec, to document that the voltage should be around 1.4 volts. You just forgot. So the first aspiring manufacturer of 100% compatible headphones comes along, his speaker is only expecting 0.014 volts, and when he tests his prototype, it either blows out the headphones, or the eardrums of the listener, whichever comes first. And he makes some adjustments and eventually gets a headphone that works fine and is just a couple of angstroms more fierce than your headphones.

More and more manufacturers show up with compatible headphones, and soon we’re in a ONE TO MANY market.

[1] to [many]

So far, all is well. We have a de-facto standard for headphone jacks here. The written spec is not complete and not adequate, but anybody who wants to make a compatible headphone just has to plug it into your personal stereo device and test it, and if it works, all is well, they can sell it, and it will work.

Until you decide to make a new version, the Qxyzrhjjjjukltk 2.0.

The Qxyzrhjjjjukltk 2.0 is going to include a telephone (turns out Marslings didn’t figure out cell phones on their own, either) and the headphone is going to have to have a built-in microphone, which requires one more conductor, so you rework the connector into something totally incompatible and kind of ugly, with all kinds of room for expansion:

Completely different 25-conductor connector

And the Qxyzrhjjjjukltk 2.0 is a complete and utter failure in the market. Yes, it has a nice telephone thing, but nobody cared about that. They cared about their large collections of headphones. It turns out that when I said Marslings are very particular about the color of things that they stick in their ears, I meant it. Most trendy Marslings at this point have a whole closet full of nice headphones. They all look the same to you (red), but Marslings are very, very finicky about shades of red in a way that you never imagined. The newest high-end apartments on Mars are being marketed with a headphone closet. I kid you not.

So the new jack is not such a success, and you quickly figure out a new scheme:

Stereo jack with extra insulating ring on shaft allowing for 4 conductors instead of 3

Notice that you’ve now split the main shaft to provide another conductor for the microphone signal, but the trouble is, your Qxyzrhjjjjukltk 2.1 doesn’t really know whether the headset that’s plugged in has a mic or not, and it needs to know this so it can decide whether to enable phone calls. And so you invent a little protocol… the new device puts a signal on the mic pin, and looks for it on the ground, and if it’s there, it must be a three conductor plug, and therefore they don’t have a mic, so you’ll go into backwards compatibility mode where you only play music. It’s simple, but it’s a protocol negotiation.

It’s not a ONE-MANY market any more. All the stereo devices are made by the same firm, one after the other, so I’m going to call this a SEQUENCE-MANY market:

[1,2,3] to [MANY]

Here are some SEQUENCE-MANY markets you already know about:

  1. Facebook | about 20,000 Facebook Apps
  2. Windows | about 1,000,000 Windows Apps
  3. Microsoft Word | about 1,000,000,000 Word documents

There are hundreds of other examples. The key thing to remember is that when a new version of the left-hand device comes out, it has to maintain auto-backwards-compatibility with all the old right-hand accessories meant to work with the old device, because those old accessories could not possibly have designed with the new product in mind. The Martian headphones are already made. You can’t go back and change them all. It’s much easier and more sensible to change the newly invented device so that it acts like an old device when confronted with an old headphone.

And because you want to make progress, adding new features and functionality, you also need a new protocol for new devices to use, and the sensible thing to do is to have both devices negotiate a little bit at the beginning to decide whether they both understand the latest protocol.

SEQUENCE-MANY is the world Microsoft grew up in.

But there’s one more twist, the MANY-MANY market.

A few years pass; you’re still selling Qxyzrhjjjjukltks like crazy; but now there are lots of Qxyzrhjjjjukltk clones on the market, like the open source FireQx, and lots of headphones, and you all keep inventing new features that require changes to the headphone jack and it’s driving the headphone makers crazy because they have to test their new designs out against every Qxyzrhjjjjukltk clone which is costly and time consuming and frankly most of them don’t have time and just get it to work on the most popular Qxyzrhjjjjukltk 5.0, and if that works, they’re happy, but of course when you plug the headphones into FireQx 3.0 lo and behold they explode in your hands because of a slight misunderstanding about some obscure thing in the spec which nobody really understands called hasLayout, and everybody understands that when it’s raining the hasLayout property is true and the voltage is supposed to increase to support the windshield-wiper feature, but there seems to be some debate over whether hail and snow are rain for the purpose of hasLayout, because the spec just doesn’t say. FireQx 3.0 treats snow as rain, because you need windshield wipers in the snow, Qxyzrhjjjjukltk 5.0 does not, because the programmer who worked on that feature lives in a warm part of Mars without snow and doesn’t have a driver’s license anyway. Yes, they have driver’s licenses on Mars.

And eventually some tedious bore writes a lengthy article on her blog explaining a trick you can use to make Qxyzrhjjjjukltk 5.0 behave just like FireQx 3.0 through taking advantage of a bug in Qxyzrhjjjjukltk 5.0 in which you trick Qxyzrhjjjjukltk into deciding that it’s raining when it’s snowing by melting a little bit of the snow, and it’s ridiculous, but everyone does it, because they have to solve the hasLayout incompatibility. Then the Qxyzrhjjjjukltk team fixes that bug in 6.0, and you’re screwed again, and you have to go find some new bug to exploit to make your windshield-wiper-equipped headphone work with either device.

NOW. This is the MANY-MANY market. Many players on the left hand side who don’t cooperate, and SCRILLIONS of players on the right hand side. And they’re all making mistakes because To Err Is Human.

[MANY] to [MANY]

And of course this is the situation we find ourselves in with HTML. Dozens of common browsers, literally billions of web pages.

[MANY (web browsers)] - [MANY (web sites)]

And over the years what happens in a MANY-MANY market is that there is a hue and cry for “standards” so that “all the players” (meaning, the small players) have an equal chance to being able to display all 8 billion web pages correctly, and, even more importantly, so that the designers of those 8 billion pages only have to test against one browser, and use “web standards,” and then they will know that their page will also work in other browsers, without having to test every page against every browser.

[MANY (web browsers)] - A SINGLE STANDARD - [MANY (web sites)]

See, the idea is, instead of many-many testing, you have many-standard and standard-many testing and you need radically fewer tests. Not to mention that your web pages don’t need any browser-specific code to work around bugs in individual browsers, because in this platonic world there are no bugs.

That’s the ideal.

In practice, with the web, there’s a bit of a problem: no way to test a web page against the standard, because there’s no reference implementation that guarantees that if it works, all the browsers work. This just doesn’t exist.

So you have to “test” in your own head, purely as a thought experiment, against a bunch of standards documents which you probably never read and couldn’t completely understand even if you did.

Those documents are super confusing. The specs are full of statements like “If a sibling block box (that does not float and is not absolutely positioned) follows the run-in box, the run-in box becomes the first inline box of the block box. A run-in cannot run in to a block that already starts with a run-in or that itself is a run-in.” Whenever I read things like that, I wonder how anyone correctly conforms to the spec.

There is no practical way to check if the web page you just coded conforms to the spec. There are validators, but they won’t tell you what the page is supposed to look like, and having a “valid” page where all the text is overlapping and nothing lines up and you can’t see anything is not very useful. What people do is check their pages against one browser, maybe two, until it looks right. And if they’ve made a mistake that just happens to look OK in IE and Firefox, they’re not even going to know about it.

And their pages may break when a future web browser comes out.

If you’ve ever visited the ultra-orthodox Jewish communities of Jerusalem, all of whom agree in complete and utter adherence to every iota of Jewish law, you will discover that despite general agreement on what constitutes kosher food, that you will not find a rabbi from one ultra-orthodox community who is willing to eat at the home of a rabbi from a different ultra-orthodox community. And the web designers are discovering what the Jews of Mea Shearim have known for decades: just because you all agree to follow one book doesn’t ensure compatibility, because the laws are so complex and complicated and convoluted that it’s almost impossible to understand them all well enough to avoid traps and landmines, and you’re safer just asking for the fruit plate.

Standards are a great goal, of course, but before you become a standards fanatic you have to understand that due to the failings of human beings, standards are sometimes misinterpreted, sometimes confusing and even ambiguous.

The precise problem here is that you’re pretending that there’s one standard, but since nobody has a way to test against the standard, it’s not a real standard: it’s a platonic ideal and a set of misinterpretations, and therefore the standard is not serving the desired goal of reducing the test matrix in a MANY-MANY market.

DOCTYPE is a myth.

A mortal web designer who attaches a DOCTYPE tag to their web page saying, “this is standard HTML,” is committing an act of hubris. There is no way they know that. All they are really saying is that the page was meant to be standard HTML. All they really know is that they tested it with IE, Firefox, maybe Opera and Safari, and it seems to work. Or, they copied the DOCTYPE tag out of a book and don’t know what it means.

In the real world where people are imperfect, you can’t have a standard with just a spec–you must have a super-strict reference implementation, and everybody has to test against the reference implementation. Otherwise you get 17 different “standards” and you might as well not have one at all.

And this is where Jon Postel caused a problem, back in 1981, when he coined the robustness principle: “Be conservative in what you do, be liberal in what you accept from others.” What he was trying to say was that the best way to make the protocols work robustly would be if everyone was very, very careful to conform to the specification, but they should be also be extremely forgiving when talking to partners that don’t conform exactly to the specification, as long as you can kind of figure out what they meant.

So, technically, the way to make a paragraph with small text is <p><small>, but a lot of people wrote <small><p> which is technically incorrect for reasons most web developers don’t understand, and the web browsers forgave them and made the text small anyway, because that’s obviously what they wanted to happen.

Now there are all these web pages out there with errors, because all the early web browser developers made super-liberal, friendly, accommodating browsers that loved you for who you were and didn’t care if you made a mistake. And so there were lots of mistakes. And Postel’s “robustness” principle didn’t really work. The problem wasn’t noticed for many years. In 2001 Marshall Rose finally wrote:

Counter-intuitively, Postel’s robustness principle (“be conservative in what you send, liberal in what you accept”) often leads to deployment problems. Why? When a new implementation is initially fielded, it is likely that it will encounter only a subset of existing implementations. If those implementations follow the robustness principle, then errors in the new implementation will likely go undetected. The new implementation then sees some, but not widespread deployment. This process repeats for several new implementations. Eventually, the not-quite-correct implementations run into other implementations that are less liberal than the initial set of implementations. The reader should be able to figure out what happens next.

Jon Postel should be honored for his enormous contributions to the invention of the Internet, and there is really no reason to fault him for the infamous robustness principle. 1981 is prehistoric. If you had told Postel that there would be 90 million untrained people, not engineers, creating web sites, and they would be doing all kinds of awful things, and some kind of misguided charity would have caused the early browser makers to accept these errors and display the page anyway, he would have understood that this is the wrong principle, and that, actually, the web standards idealists are right, and the way the web “should have” been built would be to have very, very strict standards and every web browser should be positively obnoxious about pointing them all out to you and web developers that couldn’t figure out how to be “conservative in what they emit” should not be allowed to author pages that appear anywhere until they get their act together.

But, of course, if that had happened, maybe the web would never have taken off like it did, and maybe instead, we’d all be using a gigantic Lotus Notes network operated by AT&T. Shudder.

Shoulda woulda coulda. Who cares. We are where we are. We can’t change the past, only the future. Heck, we can barely even change the future.

And if you’re a pragmatist on the Internet Explorer 8.0 team, you might have these words from Raymond Chen seared into your cortex. He was writing about how Windows XP had to emulate buggy behavior from old versions of Windows:

Look at the scenario from the customer’s standpoint. You bought programs X, Y and Z. You then upgraded to Windows XP. Your computer now crashes randomly, and program Z doesn’t work at all. You’re going to tell your friends, “Don’t upgrade to Windows XP. It crashes randomly, and it’s not compatible with program Z.” Are you going to debug your system to determine that program X is causing the crashes, and that program Z doesn’t work because it is using undocumented window messages? Of course not. You’re going to return the Windows XP box for a refund. (You bought programs X, Y, and Z some months ago. The 30-day return policy no longer applies to them. The only thing you can return is Windows XP.)

And you’re thinking, hmm, let’s update this for today:

Look at the scenario from the customer’s standpoint. You bought programs X, Y and Z. You then upgraded to Windows XPVista. Your computer now crashes randomly, and program Z doesn’t work at all. You’re going to tell your friends, “Don’t upgrade to Windows XPVista. It crashes randomly, and it’s not compatible with program Z.” Are you going to debug your system to determine that program X is causing the crashes, and that program Z doesn’t work because it is using undocumentedinsecure window messages? Of course not. You’re going to return the Windows XPVista box for a refund. (You bought programs X, Y, and Z some months ago. The 30-day return policy no longer applies to them. The only thing you can return is Windows XPVista.)

The victory of the idealists over the pragmatists at Microsoft, which I reported in 2004, directly explains why Vista is getting terrible reviews and selling poorly.

And how does it apply to the IE team?

Look at the scenario from the customer’s standpoint. You visit 100 websites a day. You then upgraded to IE 8. On half of them, the page is messed up, and Google Maps doesn’t work at all.

You’re going to tell your friends, “Don’t upgrade to IE 8. It messes up every page, and Google Maps doesn’t work at all.” Are you going to View Source to determine that website X is using nonstandard HTML, and Google Maps doesn’t work because it is using non-standard JavaScript objects from old versions of IE that were never accepted by the standards committee? Of course not. You’re going to uninstall IE 8. (Those websites are out of your control. Some of them were developed by people who are now dead. The only thing you can do is go back to IE 7).

And so if you’re a developer on the IE 8 team, your first inclination is going to be to do exactly what has always worked in these kinds of SEQUENCE-MANY markets. You’re going to do a little protocol negotiation, and continue to emulate the old behavior for every site that doesn’t explicitly tell you that they expect the new behavior, so that all existing web pages continue to work, and you’re only going to have the nice new behavior for sites that put a little flag on the page saying, “Yo! I grok IE 8! Give me all the new IE 8 Goodness Please!”

And indeed that was the first decision announced by the IE team on January 21st. The web browser would accommodate existing pages silently so that nobody had to change their web site by acting like the old, buggy IE7 that web developers hated.

A pragmatic engineer would have to come to the conclusion that the IE team’s first decision was right. But the young idealist “standards” people went nuclear.

IE needed to provide a web standards experience without requiring a special “Yo! I’m tested with IE 8!” tag, they said. They were sick of special tags. Every frigging web page has to have thirty seven ugly hacks in it to make it work with five or six popular browsers. Enough ugly hacks. 8 billion existing web pages be damned.

And the IE team flip-flopped. Their second decision, and I have to think it’s not final, their second decision was to do the idealistic thing, and treat all sites that claim to be “standards-compliant” as if they have been designed for and tested with IE8.

Almost every web site I visited with IE8 is broken in some way. Websites that use a lot of JavaScript are generally completely dead. A lot of pages simply have visual problems: things in the wrong place, popup menus that pop under, mysterious scrollbars in the middle. Some sites have more subtle problems: they look ok but as you go further you find that critical form won’t submit or leads to a blank page.

These are not web pages with errors. They are usually websites which were carefully constructed to conform to web standards. But IE 6 and IE 7 didn’t really conform to the specs, so these sites have little hacks in them that say, “on Internet Explorer… move this thing 17 pixels to the right to compensate for IE’s bug.”

And IE 8 is IE, but it no longer has the IE 7 bug where it moved that thing 17 pixels left of where it was supposed to be according to web standards. So now code that was written that was completely reasonable no longer works.

IE 8 can’t display most web pages correctly until you give up and press the “ACT LIKE IE7″ button. The idealists don’t care: they want those pages changed.

Some of those pages can’t be changed. They might be burned onto CD-ROMs. Some of them were created by people who are now dead. Most of them created by people who have no frigging idea what’s going on and why their web page, which they paid a designer to create 4 years ago, is now not working properly.

The idealists rejoiced. Hundreds of them descended on the IE blog to actually say nice things about Microsoft for the first times in their lives.

I looked at my watch.

Tick, tick, tick.

Within a matter of seconds, you started to see people on the forums showing up like this one:

I have downloaded IE 8 and with it some bugs. Some of my websites like “HP” are very difficult to read as the whole page is very very small… The speed of my Internet has also been reduced on some occasions. Whe I use Google Maps, there are overlays everywhere, enough so it makes it ackward to use!

Mmhmm. All you smug idealists are laughing at this newbie/idjit. The consumer is not an idiot. She’s your wife. So stop laughing. 98% of the world will install IE8 and say, “It has bugs and I can’t see my sites.” They don’t give a flicking flick about your stupid religious enthusiasm for making web browsers which conform to some mythical, platonic “standard” that is not actually implemented anywhere. They don’t want to hear your stories about messy hacks. They want web browsers that work with actual web sites.

So you see, we have a terrific example here of a gigantic rift between two camps.

The web standards camp seems kind of Trotskyist. You’d think they’re the left wing, but if you happened to make a website that claims to conform to web standards but doesn’t, the idealists turn into Joe Arpaio, America’s Toughest Sheriff. “YOU MADE A MISTAKE AND YOUR WEBSITE SHOULD BREAK. I don’t care if 80% of your websites stop working. I’ll put you all in jail, where you will wear pink pajamas and eat 15 cent sandwiches and work on a chain gang. And I don’t care if the whole county is in jail. The law is the law.”

On the other hand, we have the pragmatic, touchy feely, warm and fuzzy engineering types. “Can’t we just default to IE7 mode? One line of code … Zip! Solved!”

Secretly? Here’s what I think is going to happen. The IE8 team going to tell everyone that IE8 will use web standards by default, and run a nice long beta during which they beg people to test their pages with IE8 and get them to work. And when they get closer to shipping, and only 32% of the web pages in the world render properly, they’ll say, “look guys, we’re really sorry, we really wanted IE8 standards mode to be the default, but we can’t ship a browser that doesn’t work,” and they’ll revert to the pragmatic decision. Or maybe they won’t, because the pragmatists at Microsoft have been out of power for a long time. In which case, IE is going to lose a lot of market share, which would please the idealists to no end, and probably won’t decrease Dean Hachamovitch’s big year-end bonus by one cent.

You see? No right answer.

As usual, the idealists are 100% right in principle and, as usual, the pragmatists are right in practice. The flames will continue for years. This debate precisely splits the world in two. If you have a way to buy stock in Internet flame wars, now would be a good time to do that.

Strategy Letter VI

IBM just released an open-source office suite called IBM Lotus Symphony. Sounds like Yet Another StarOffice distribution. But I suspect they’re probably trying to wipe out the memory of the original Lotus Symphony, which had been hyped as the Second Coming and which fell totally flat. It was the software equivalent of Gigli.

In the late 80s, Lotus was trying very hard to figure out what to do next with their flagship spreadsheet and graphics product, Lotus 1-2-3. There were two obvious ideas: first, they could add more features. Word processing, say. This product was called Symphony. Another idea which seemed obvious was to make a 3-D spreadsheet. That became 1-2-3 version 3.0.

Both ideas ran head-first into a serious problem: the old DOS 640K memory limitation. IBM was starting to ship a few computers with 80286 chips, which could address more memory, but Lotus didn’t think there was a big enough market for software that needed a $10,000 computer to run. So they squeezed and squeezed. They spent 18 months cramming 1-2-3 for DOS into 640K, and eventually, after a lot of wasted time, had to give up the 3D feature to get it to fit. In the case of Symphony, they just chopped features left and right.

Neither strategy was right. By the time 123 3.0 was shipping, everybody had 80386s with 2M or 4M of RAM. And Symphony had an inadequate spreadsheet, an inadequate word processor, and some other inadequate bits.

“That’s nice, old man,” you say. “Who gives a fart about some old character mode software?”

Humor me for a minute, because history is repeating itself, in three different ways, and the smart strategy is to bet on the same results.

Limited-memory, limited-CPU environments

From the beginning of time until about, say, 1989, programmers were extremely concerned with efficiency. There just wasn’t that much memory and there just weren’t that many CPU cycles.

In the late 90s a couple of companies, including Microsoft and Apple, noticed (just a little bit sooner than anyone else) that Moore’s Law meant that they shouldn’t think too hard about performance and memory usage… just build cool stuff, and wait for the hardware to catch up. Microsoft first shipped Excel for Windows when 80386s were too expensive to buy, but they were patient. Within a couple of years, the 80386SX came out, and anybody who could afford a $1500 clone could run Excel.

As a programmer, thanks to plummeting memory prices, and CPU speeds doubling every year, you had a choice. You could spend six months rewriting your inner loops in Assembler, or take six months off to play drums in a rock and roll band, and in either case, your program would run faster. Assembler programmers don’t have groupies.

So, we don’t care about performance or optimization much anymore.

Except in one place: JavaScript running on browsers in AJAX applications. And since that’s the direction almost all software development is moving, that’s a big deal.

A lot of today’s AJAX applications have a meg or more of client side code. This time, it’s not the RAM or CPU cycles that are scarce: it’s the download bandwidth and the compile time. Either way, you really have to squeeze to get complex AJAX apps to perform well.

History, though, is repeating itself. Bandwidth is getting cheaper. People are figuring out how to precompile JavaScript.

The developers who put a lot of effort into optimizing things and making them tight and fast will wake up to discover that effort was, more or less, wasted, or, at the very least, you could say that it “conferred no long term competitive advantage,” if you’re the kind of person who talks like an economist.

The developers who ignored performance and blasted ahead adding cool features to their applications will, in the long run, have better applications.

A portable programming language

The C programming language was invented with the explicit goal of making it easy to port applications from one instruction set to another. And it did a fine job, but wasn’t really 100% portable, so we got Java, which was even more portable than C. Mmmhmm.

Right now the big hole in the portability story is — tada! — client-side JavaScript, and especially the DOM in web browsers. Writing applications that work in all different browsers is a friggin’ nightmare. There is simply no alternative but to test exhaustively on Firefox, IE6, IE7, Safari, and Opera, and guess what? I don’t have time to test on Opera. Sucks to be Opera. Startup web browsers don’t stand a chance.

What’s going to happen? Well, you can try begging Microsoft and Firefox to be more compatible. Good luck with that. You can follow the p-code/Java model and build a little sandbox on top of the underlying system. But sandboxes are penalty boxes; they’re slow and they suck, which is why Java Applets are dead, dead, dead. To build a sandbox you pretty much doom yourself to running at 1/10th the speed of the underlying platform, and you doom yourself to never supporting any of the cool features that show up on one of the platforms but not the others. (I’m still waiting for someone to show me a Java applet for phones that can access any of the phone’s features, like the camera, the contacts list, the SMS messages, or the GPS receiver.)

Sandboxes didn’t work then and they’re not working now.

What’s going to happen? The winners are going to do what worked at Bell Labs in 1978: build a programming language, like C, that’s portable and efficient. It should compile down to “native” code (native code being JavaScript and DOMs) with different backends for different target platforms, where the compiler writers obsess about performance so you don’t have to. It’ll have all the same performance as native JavaScript with full access to the DOM in a consistent fashion, and it’ll compile down to IE native and Firefox native portably and automatically. And, yes, it’ll go into your CSS and muck around with it in some frightening but provably-correct way so you never have to think about CSS incompatibilities ever again. Ever. Oh joyous day that will be.

High interactivity and UI standards

The IBM 360 mainframe computer system used a user interface called CICS, which you can still see at the airport if you lean over the checkin counter. There’s an 80 character by 24 character green screen, character mode only, of course. The mainframe sends down a form to the “client” (the client being a 3270 smart terminal). The terminal is smart; it knows how to present the form to you and let you input data into the form without talking to the mainframe at all. This was one reason mainframes were so much more powerful than Unix: the CPU didn’t have to handle your line editing; it was offloaded to a smart terminal. (If you couldn’t afford smart terminals for everyone, you bought a System/1 minicomputer to sit between the dumb terminals and the mainframe and handle the form editing for you).

Anyhoo, after you filled out your form, you pressed SEND, and all your answers were sent back to the server to process. Then it sent you another form. And on and on.

Awful. How do you make a word processor in that kind of environment? (You really can’t. There never was a decent word processor for mainframes).

That was the first stage. It corresponds precisely to the HTML phase of the Internet. HTML is CICS with fonts.

In the second stage, everybody bought PCs for their desks, and suddenly, programmers could poke text anywhere on the screen wily-nily, anywhere they wanted, any time they wanted, and you could actually read every keystroke from the users as they typed, so you could make a nice fast application that didn’t have to wait for you to hit SEND before the CPU could get involved. So, for example, you could make a word processor that automatically wrapped, moving a word down to the next line when the current line filled up. Right away. Oh my god. You can do that?

The trouble with the second stage was that there were no clear UI standards… the programmers almost had too much flexibility, so everybody did things in different ways, which made it hard, if you knew how to use program X, to also use program Y. WordPerfect and Lotus 1-2-3 had completely different menu systems, keyboard interfaces, and command structures. And copying data between them was out of the question.

And that’s exactly where we are with Ajax development today. Sure, yeah, the usability is much better than the first generation DOS apps, because we’ve learned some things since then. But Ajax apps can be inconsistent, and have a lot of trouble working together — you can’t really cut and paste objects from one Ajax app to another, for example, so I’m not sure how you get a picture from Gmail to Flickr. Come on guys, Cut and Paste was invented 25 years ago.

The third phase with PCs was Macintosh and Windows. A standard, consistent user interface with features like multiple windows and the Clipboard designed so that applications could work together. The increased usability and power we got out of the new GUIs made personal computing explode.

So if history repeats itself, we can expect some standardization of Ajax user interfaces to happen in the same way we got Microsoft Windows. Somebody is going to write a compelling SDK that you can use to make powerful Ajax applications with common user interface elements that work together. And whichever SDK wins the most developer mindshare will have the same kind of competitive stronghold as Microsoft had with their Windows API.

If you’re a web app developer, and you don’t want to support the SDK everybody else is supporting, you’ll increasingly find that people won’t use your web app, because it doesn’t, you know, cut and paste and support address book synchronization and whatever weird new interop features we’ll want in 2010.

Imagine, for example, that you’re Google with GMail, and you’re feeling rather smug. But then somebody you’ve never heard of, some bratty Y Combinator startup, maybe, is gaining ridiculous traction selling NewSDK, which combines a great portable programming language that compiles to JavaScript, and even better, a huge Ajaxy library that includes all kinds of clever interop features. Not just cut ‘n’ paste: cool mashup features like synchronization and single-point identity management (so you don’t have to tell Facebook and Twitter what you’re doing, you can just enter it in one place). And you laugh at them, for their NewSDK is a honking 232 megabytes … 232 megabytes! … of JavaScript, and it takes 76 seconds to load a page. And your app, GMail, doesn’t lose any customers.

But then, while you’re sitting on your googlechair in the googleplex sipping googleccinos and feeling smuggy smug smug smug, new versions of the browsers come out that support cached, compiled JavaScript. And suddenly NewSDK is really fast. And Paul Graham gives them another 6000 boxes of instant noodles to eat, so they stay in business another three years perfecting things.

And your programmers are like, jeez louise, GMail is huge, we can’t port GMail to this stupid NewSDK. We’d have to change every line of code. Heck it’d be a complete rewrite; the whole programming model is upside down and recursive and the portable programming language has more parentheses than even Google can buy. The last line of almost every function consists of a string of 3,296 right parentheses. You have to buy a special editor to count them.

And the NewSDK people ship a pretty decent word processor and a pretty decent email app and a killer Facebook/Twitter event publisher that synchronizes with everything, so people start using it.

And while you’re not paying attention, everybody starts writing NewSDK apps, and they’re really good, and suddenly businesses ONLY want NewSDK apps, and all those old-school Plain Ajax apps look pathetic and won’t cut and paste and mash and sync and play drums nicely with one another. And Gmail becomes a legacy. The WordPerfect of Email. And you’ll tell your children how excited you were to get 2GB to store email, and they’ll laugh at you. Their nail polish has more than 2GB.

Crazy story? Substitute “Google Gmail” with “Lotus 1-2-3”. The NewSDK will be the second coming of Microsoft Windows; this is exactly how Lotus lost control of the spreadsheet market. And it’s going to happen again on the web because all the same dynamics and forces are in place. The only thing we don’t know yet are the particulars, but it’ll happen.

Seven steps to remarkable customer service

As a bootstrapped software company, Fog Creek couldn’t afford to hire customer service people for the first couple of years, so Michael and I did it ourselves. The time we spent helping customers took away from improving our software, but we learned a lot and now we have a much better customer service operation.

Here are seven things we learned about providing remarkable customer service. I’m using the word remarkable literally—the goal is to provide customer service so good that people remark.

1. Fix everything two ways

Almost every tech support problem has two solutions. The superficial and immediate solution is just to solve the customer’s problem. But when you think a little harder you can usually find a deeper solution: a way to prevent this particular problem from ever happening again.

Sometimes that means adding more intelligence to the software or the SETUP program; by now, our SETUP program is loaded with special case checks. Sometimes you just need to improve the wording of an error message. Sometimes the best you can come up with is a knowledge base article.

We treat each tech support call like the NTSB treats airliner crashes. Every time a plane crashes, they send out investigators, figure out what happened, and then figure out a new policy to prevent that particular problem from ever happening again. It’s worked so well for aviation safety that the very, very rare airliner crashes we still get in the US are always very unusual, one-off situations.

This has two implications.

One: it’s crucial that tech support have access to the development team. This means that you can’t outsource tech support: they have to be right there at the same street address as the developers, with a way to get things fixed. Many software companies still think that it’s “economical” to run tech support in Bangalore or the Philippines, or to outsource it to another company altogether. Yes, the cost of a single incident might be $10 instead of $50, but you’re going to have to pay $10 again and again.

When we handle a tech support incident with a well-qualified person here in New York, chances are that’s the last time we’re ever going to see that particular incident. So with one $50 incident we’ve eliminated an entire class of problems.

Somehow, the phone companies and the cable companies and the ISPs just don’t understand this equation. They outsource their tech support to the cheapest possible provider and end up paying $10 again and again and again fixing the same problem again and again and again instead of fixing it once and for all in the source code. The cheap call centers have no mechanism for getting problems fixed; indeed, they have no incentive to get problems fixed because their income depends on repeat business, and there’s nothing they like better than being able to give the same answer to the same question again and again.

The second implication of fixing everything two ways is that eventually, all the common and simple problems are solved, and what you’re left with is very weird uncommon problems. That’s fine, because there are far fewer of them, and you’re saving a fortune not doing any rote tech support, but the downside is that there’s no rote tech support left: only serious debugging and problem solving. You can’t just teach new support people ten common solutions: you have to teach them to debug.

For us, the “fix everything two ways” religion has really paid off. We were able to increase our sales tenfold while only doubling the cost of providing tech support.

2. Suggest blowing out the dust

Microsoft’s Raymond Chen tells the story of a customer who complains that the keyboard isn’t working. Of course, it’s unplugged. If you try asking them if it’s plugged in, “they will get all insulted and say indignantly, ‘Of course it is! Do I look like an idiot?’ without actually checking.”

“Instead,” Chen suggests, “say ‘Okay, sometimes the connection gets a little dusty and the connection gets weak. Could you unplug the connector, blow into it to get the dust out, then plug it back in?’

“They will then crawl under the desk, find that they forgot to plug it in (or plugged it into the wrong port), blow out the dust, plug it in, and reply, ‘Um, yeah, that fixed it, thanks.’”

Many requests for a customer to check something can be phrased this way. Instead of telling them to check a setting, tell them to change the setting and then change it back “just to make sure that the software writes out its settings.”

3. Make customers into fans

Every time we need to buy logo gear here at Fog Creek, I get it from Lands’ End.


Let me tell you a story. We needed some shirts for a trade show. I called up Lands’ End and ordered two dozen, using the same logo design we had used for some knapsacks we bought earlier.

When the shirts arrived, to our dismay, you couldn’t read the logo.

It turns out that the knapsacks were brighter than the polo shirts. The thread color that looked good on the knapsacks was too dark to read on the shirts.

I called up Lands’ End. As usual, a human answered the phone even before it started ringing. I’m pretty sure that they have a system where the next agent in the queue is told to standby, so customers don’t even have to wait one ringy-dingy before they’re talking to a human.

I explained that I screwed up.

They said, “Don’t worry. You can return those for a full credit, and we’ll redo the shirts with a different color thread.”

I said, “The trade show is in two days.”

They said they would Fedex me a new box of shirts and I’d have it tomorrow. I could return the old shirts at my convenience.

They paid shipping both ways. I wasn’t out a cent. Even though they had no possible use for a bunch of Fog Creek logo shirts with an illegible logo, they ate the cost.

And now I tell this story to everyone who needs swag. In fact I tell this story every time we’re talking about telephone menu systems. Or customer service. By providing remarkable customer service, they’ve gotten me to remark about it.

When customers have a problem and you fix it, they’re actually going to be even more satisfied than if they never had a problem in the first place.

It has to do with expectations. Most people’s experience with tech support and customer service comes from airlines, telephone companies, cable companies, and ISPs, all of whom provide generally awful customer service. It’s so bad you don’t even bother calling any more, do you? So when someone calls Fog Creek, and immediately gets through to a human, with no voice mail or phone menus, and that person turns out to be nice and friendly and actually solves their problem, they’re apt to think even more highly of us than someone who never had the opportunity to interact with us and just assumes that we’re average.

Now, I wouldn’t go so far as to actually make something go wrong, just so we have a chance to demonstrate our superior customer service. Many customers just won’t call; they’ll fume quietly.

But when someone does call, look at it as a great opportunity to create fanatically devoted customer, one who will prattle on and on about what a great job you did.

4. Take the blame

One morning I needed an extra set of keys to my apartment, so on the way to work, I went to the locksmith around the corner.

13 years living in an apartment in New York City has taught me never to trust a locksmith; half of the time their copies don’t work. So I went home to test the new keys, and, lo and behold, one didn’t work.

I took it back to the locksmith.

He made it again.

I went back home and tested the new copy.

It still didn’t work.

Now I was fuming. Squiggly lines were coming up out of my head. I was a half hour late to work and had to go to the locksmith for a third time. I was tempted just to give up on him. But I decided to give this loser one more chance.

I stomped into the store, ready to unleash my fury.

“It still doesn’t work?” he asked. “Let me see.”

He looked at it.

I was sputtering, trying to figure out how best to express my rage at being forced to spend the morning going back and forth.

“Ah. It’s my fault,” he said.

And suddenly, I wasn’t mad at all.

Mysteriously, the words “it’s my fault” completely defused me. That was all it took.

He made the key a third time. I wasn’t mad any more. The key worked.

And, here I was, on this planet for forty years, and I couldn’t believe how much the three words “it’s my fault” had completely changed my emotions in a matter of seconds.

Most locksmiths in New York are not the kinds of guys to admit that they’re wrong. Saying “it’s my fault” was completely out of character. But he did it anyway.

5. Memorize awkward phrases

I figured, OK, since the morning is shot anyway, I might as well go to the diner for some breakfast.

It’s one of those classic New York diners, like the one on Seinfeld. There’s a thirty page menu and a kitchen the size of a phone booth. It doesn’t make sense. They must have Star Trek technology to get all those ingredients into such a small space. Maybe they rearrange atoms on the spot.

I was sitting by the cash register.

An older woman came up to pay her check. As she was paying, she said to the owner, “you know, I’ve been coming here for years and years, and that waiter was really rather rude to me.”

The owner was furious.

“What do you mean? No he wasn’t! He’s a good waiter! I never had a complaint!’

The customer couldn’t believe it. Here she was, a loyal customer, and she wanted to help out the owner by letting him know that one of his waiters needed a little bit of help in the manners department, but the owner was arguing with her!

“Well, that’s fine, but I’ve been coming here for years, and everybody is always very nice to me, but that guy was rude to me,” she explained, patiently.

“I don’t care if you’ve been coming here forever. My waiters are not rude.” The owner proceeded to yell at her. “I never had no problems. Why are you making problems?”

“Look, if you’re going to treat me this way I won’t come back.”

“I don’t care!” said the owner. One of the great things about owning a diner in New York is that there are so many people in the city that you can offend every single customer who ever comes into your diner and you’ll still have a lot of customers. “Don’t come back! I don’t want you as a customer!”

Good for you, I thought. Here’s a 60-something year old man, owner of a diner, and you won some big moral victory against a little old lady. Are you proud of yourself? How macho do you have to be? Does the moral victory make you feel better? Did you really have to lose a repeat customer?

Would it have made you feel totally emasculated to say, “I’m so sorry. I’ll have a word with him?”

It’s easy to get caught up in the emotional heat of the moment when someone is complaining.

The solution is to memorize some key phrases, and practice saying them, so that when you need to say them, you can forget your testosterone and make a customer happy.

“I’m sorry, it’s my fault.”

“I’m sorry, I can’t accept your money. The meal’s on me.”

“That’s terrible, please tell me what happened so I can make sure it never happens again.”

It’s completely natural to have trouble saying “It’s my fault.” That’s human. But those three words are going to make your angry customers much happier. So you’re going to have to say them. And you’re going to have to sound like you mean it.

So start practicing.

Say “It’s my fault” a hundred times one morning in the shower, until it starts to sound like syllabic nonsense. Then you’ll be able to say it on demand.

One more point. You may think that admitting fault is a strict no-no that can get you sued. This is nonsense. The way to avoid getting sued is not to have people who are mad at you. The best way to do this is to admit fault and fix the damn problem.

6. Practice puppetry

The angry diner owner clearly took things very personally, in a way that the locksmith didn’t. When an irate customer is complaining, or venting, it’s easy to get defensive.

You can never win these arguments, and if you take them personally, it’s going to be a million times worse. This is when you start to hear business owners saying, “I don’t want an asshole like you for a customer!” They get excited about their Pyrrhic victory. Wow, isn’t it great? When you’re a small business owner you get to fire your customers. Charming.

The bottom line is that this is not good for business, and it’s not even good for your emotional well-being. When you win a victory with a customer by firing them, you still end up feeling riled up and angry, they’ll get their money back from the credit card company anyway, and they’ll tell a dozen friends. As Patrick McKenzie writes, “You will never win an argument with your customer.”

There is only one way to survive angry customers emotionally: you have to realize that they’re not angry at you; they’re angry at your business, and you just happen to be a convenient representative of that business.

And since they’re treating you like a puppet, an iconic stand-in for the real business, you need to treat yourself as a puppet, too.

Pretend you’re a puppeteer. The customer is yelling at the puppet. They’re not yelling at you. They’re angry with the puppet.

Your job is to figure out, “gosh, what can I make the puppet say that will make this person a happy customer?”

You’re just a puppeteer. You’re not a party to the argument. When the customer says, “what the hell is wrong with you people,” they’re just playing a role (in this case, they’re quoting Tom Smykowski in the movie Office Space). You, too, get to play a role. “I’m sorry. It’s my fault.” Figure out what to make the puppet do that will make them happy and stop taking it so dang personally.

7. Greed will get you nowhere

Recently I was talking with the people who have been doing most of the customer service for Fog Creek over the last year, and I asked what methods they found most effective for dealing with angry customers.

“Frankly,” they said, “we have pretty nice customers. We haven’t really had any angry customers.”

Well, OK, we do have nice customers, but it seems rather unusual that in a year of answering the phones, nobody was angry. I thought the nature of working at a call center was dealing with angry people all day long.

“Nope. Our customers are nice.”

Here’s what I think. I think that our customers are nice because they’re not worried. They’re not worried because we have a ridiculously liberal return policy: “We don’t want your money if you’re not amazingly happy.”

Customers know that they have nothing to fear. They have the power in the relationship. So they don’t get abusive.

The no-questions-asked 90-day money back guarantee was one of the best decisions we ever made at Fog Creek. Try this: use Fog Creek Copilot for a full 24 hours, call up three months later and say, “hey guys, I need $5 for a cup of coffee. Give me back my money from that Copilot day pass,” and we’ll give it back to you. Try calling on the 91st or 92nd or 203rd day. You’ll still get it back. We really don’t want your money if you’re not satisfied. I’m pretty sure we’re running the only job listing service around that will refund your money just because your ad didn’t work. This is unheard of, but it means we get a lot more ad listings, because there’s nothing to lose.

Over the last six years or so, letting people return software has cost us 2%.


And you know what? Most customers pay with credit cards, and if we didn’t refund their money, a bunch of them would have called their bank. This is called a chargeback. They get their money back, we pay a chargeback fee, and if this happens too often, our processing fees go up.

Know what our chargeback rate is at Fog Creek?


I’m not kidding.

If we were tougher about offering refunds, the only thing we would possibly have done is pissed a few customers off, customers who would have ranted and whined on their blogs. We wouldn’t even have kept more of their money.

I know of software companies who are very explicit on their web site that you are not entitled to a refund under any circumstances, but the truth is, if you call them up, they will eventually return your money because they know that if they don’t, your credit card company will. This is the worst of both worlds. You end up refunding the money anyway, and you don’t get to give potential customers the warm and fuzzy feeling of knowing Nothing Can Possibly Go Wrong, so they hesitate before buying. Or they don’t buy at all.

8. (Bonus!) Give customer service people a career path

The last important lesson we learned here at Fog Creek is that you need very highly qualified people talking to customers. A salesperson at Fog Creek needs to have significant experience with the software development process and needs to be able to explain why FogBugz works the way it does, and why it makes software development teams function better. A tech support person at Fog Creek can’t get by on canned answers to common questions, because we’ve eliminated the common questions by fixing the software, so tech support here has to actually troubleshoot which often means debugging.

Many qualified people get bored with front line customer service, and I’m OK with that. To compensate for this, I don’t hire people into those positions without an explicit career path. Here at Fog Creek, customer support is just the first year of a three-year management training program that includes a master’s degree in technology management at Columbia University. This allows us to get ambitious, smart geeks on a terrific career path talking to customers and solving their problems. We end up paying quite a bit more than average for these positions (especially when you consider $25,000 a year in tuition), but we get far more value out of them, too.

How Microsoft Lost the API War

Here’s a theory you hear a lot these days: “Microsoft is finished. As soon as Linux makes some inroads on the desktop and web applications replace desktop applications, the mighty empire will topple.”

Although there is some truth to the fact that Linux is a huge threat to Microsoft, predictions of the Redmond company’s demise are, to say the least, premature. Microsoft has an incredible amount of cash money in the bank and is still incredibly profitable. It has a long way to fall. It could do everything wrong for a decade before it started to be in remote danger, and you never know… they could reinvent themselves as a shaved-ice company at the last minute. So don’t be so quick to write them off. In the early 90s everyone thought IBM was completely over: mainframes were history! Back then, Robert X. Cringely predicted that the era of the mainframe would end on January 1, 2000 when all the applications written in COBOL would seize up, and rather than fix those applications, for which, allegedly, the source code had long since been lost, everybody would rewrite those applications for client-server platforms.

Well, guess what. Mainframes are still with us,  nothing happened on January 1, 2000, and IBM reinvented itself as a big ol’ technology consulting company that also happens to make cheap plastic telephones. So extrapolating from a few data points to the theory that Microsoft is finished is really quite a severe exaggeration.

However, there is a less understood phenomenon which is going largely unnoticed: Microsoft’s crown strategic jewel, the Windows API, is lost. The cornerstone of Microsoft’s monopoly power and incredibly profitable Windows and Office franchises, which account for virtually all of Microsoft’s income and covers up a huge array of unprofitable or marginally profitable product lines, the Windows API  is no longer of much interest to developers. The goose that lays the golden eggs is not quite dead, but it does have a terminal disease, one that nobody noticed yet.

Now that I’ve said that, allow me to apologize for the grandiloquence and pomposity of that preceding paragraph. I think I’m starting to sound like those editorial writers in the trade rags who go on and on about Microsoft’s strategic asset, the Windows API. It’s going to take me a few pages, here, to explain what I’m really talking about and justify my arguments. Please don’t jump to any conclusions until I explain what I’m talking about. This will be a long article. I need to explain what the Windows API is; I need to demonstrate why it’s the most important strategic asset to Microsoft; I need to explain how it was lost and what the implications of that are in the long term. And because I’m talking about big trends, I need to exaggerate and generalize.

Developers, Developers, Developers, Developers

Remember the definition of an operating system? It’s the thing that manages a computer’s resources so that application programs can run. People don’t really care much about operating systems; they care about those application programs that the operating system makes possible. Word Processors. Instant Messaging. Email. Accounts Payable. Web sites with pictures of Paris Hilton. By itself, an operating system is not that useful. People buy operating systems because of the useful applications that run on it. And therefore the most useful operating system is the one that has the most useful applications.

The logical conclusion of this is that if you’re trying to sell operating systems, the most important thing to do is make software developers want to develop software for your operating system. That’s why Steve Ballmer was jumping around the stage shouting “Developers, developers, developers, developers.” It’s so important for Microsoft that the only reason they don’t outright give away development tools for Windows is because they don’t want to inadvertently cut off the oxygen to competitive development tools vendors (well, those that are left) because having a variety of development tools available for their platform makes it that much more attractive to developers. But they really want to give away the development tools. Through their Empower ISV program you can get five complete sets of MSDN Universal (otherwise known as “basically every Microsoft product except Flight Simulator“) for about $375. Command line compilers for the .NET languages are included with the free .NET runtime… also free. The C++ compiler is now free. Anything to encourage developers to build for the .NET platform, and holding just short of wiping out companies like Borland.

Why Apple and Sun Can’t Sell Computers

Well, of course, that’s a little bit silly: of course Apple and Sun can sell computers, but not to the two most lucrative markets for computers, namely, the corporate desktop and the home computer. Apple is still down there in the very low single digits of market share and the only people with Suns on their desktops are at Sun. (Please understand that I’m talking about large trends here, and therefore when I say things like “nobody” I really mean “fewer than 10,000,000 people,” and so on and so forth.)

Why? Because Apple and Sun computers don’t run Windows programs, or, if they do, it’s in some kind of expensive emulation mode that doesn’t work so great. Remember, people buy computers for the applications that they run, and there’s so much more great desktop software available for Windows than Mac that it’s very hard to be a Mac user.

Sidebar What is this “API” thing?

If you’re writing a program, say, a word processor, and you want to display a menu, or write a file, you have to ask the operating system to do it for you, using a very specific set of function calls which are different on every operating system. These function calls are called the API: it’s the interface that an operating system, like Windows, provides to application developers, like the programmers building word processors and spreadsheets and whatnot. It’s a set of thousands and thousands of detailed and fussy functions and subroutines that programmers can use, which cause the operating system to do interesting things like display a menu, read and write files, and more esoteric things like find out how to spell out a given date in Serbian, or extremely complex things like display a web page in a window. If your program uses the API calls for Windows, it’s not going to work on Linux, which has different API calls. Sometimes they do approximately the same thing. That’s one important reason Windows software doesn’t run on Linux. If you wanted to get a Windows program to run under Linux, you’d have to reimplement the entire Windows API, which consists of thousands of complicated functions: this is almost as much work as implementing Windows itself, something which took Microsoft thousands of person-years. And if you make one tiny mistake or leave out one function that an application needs, that application will crash.

And that’s why the Windows API is such an important asset to Microsoft.

(I know, I know, at this point the 2.3% of the world that uses Macintoshes are warming up their email programs to send me a scathing letter about how much they love their Macs. Once again, I’m speaking in large trends and generalizing, so don’t waste your time. I know you love your Mac. I know it runs everything you need. I love you, you’re a Pepper, but you’re only 2.3% of the world, so this article isn’t about you.)

The Two Forces at Microsoft

There are two opposing forces inside Microsoft, which I will refer to, somewhat tongue-in-cheek, as The Raymond Chen Camp and The MSDN Magazine Camp.

Raymond Chen is a developer on the Windows team at Microsoft. He’s been there since 1992, and his weblog The Old New Thing is chock-full of detailed technical stories about why certain things are the way they are in Windows, even silly things, which turn out to have very good reasons.

The most impressive things to read on Raymond’s weblog are the stories of the incredible efforts the Windows team has made over the years to support backwards compatibility:

Look at the scenario from the customer’s standpoint. You bought programs X, Y and Z. You then upgraded to Windows XP. Your computer now crashes randomly, and program Z doesn’t work at all. You’re going to tell your friends, “Don’t upgrade to Windows XP. It crashes randomly, and it’s not compatible with program Z.” Are you going to debug your system to determine that program X is causing the crashes, and that program Z doesn’t work because it is using undocumented window messages? Of course not. You’re going to return the Windows XP box for a refund. (You bought programs X, Y, and Z some months ago. The 30-day return policy no longer applies to them. The only thing you can return is Windows XP.)

I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it.

This was not an unusual case. The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented functions or rely on buggy behavior that happens to be buggy in Windows n but is no longer buggy in Windows n+1. In fact if you poke around in the AppCompatibility section of your registry you’ll see a whole list of applications that Windows treats specially, emulating various old bugs and quirky behaviors so they’ll continue to work. Raymond Chen writes, “I get particularly furious when people accuse Microsoft of maliciously breaking applications during OS upgrades. If any application failed to run on Windows 95, I took it as a personal failure. I spent many sleepless nights fixing bugs in third-party programs just so they could keep running on Windows 95.”

A lot of developers and engineers don’t agree with this way of working. If the application did something bad, or relied on some undocumented behavior, they think, it should just break when the OS gets upgraded. The developers of the Macintosh OS at Apple have always been in this camp. It’s why so few applications from the early days of the Macintosh still work. For example, a lot of developers used to try to make their Macintosh applications run faster by copying pointers out of the jump table and calling them directly instead of using the interrupt feature of the processor like they were supposed to. Even though somewhere in Inside Macintosh, Apple’s official Bible of Macintosh programming, there was a tech note saying “you can’t do this,” they did it, and it worked, and their programs ran faster… until the next version of the OS came out and they didn’t run at all. If the company that made the application went out of business (and most of them did), well, tough luck, bubby.

To contrast, I’ve got DOS applications that I wrote in 1983 for the very original IBM PC that still run flawlessly, thanks to the Raymond Chen Camp at Microsoft. I know, it’s not just Raymond, of course: it’s the whole modus operandi of the core Windows API team. But Raymond has publicized it the most through his excellent website The Old New Thing so I’ll name it after him.

That’s one camp. The other camp is what I’m going to call the MSDN Magazine camp, which I will name after the developer’s magazine full of exciting articles about all the different ways you can shoot yourself in the foot by using esoteric combinations of Microsoft products in your own software. The MSDN Magazine Camp is always trying to convince you to use new and complicated external technology like COM+, MSMQ, MSDE, Microsoft Office, Internet Explorer and its components, MSXML, DirectX (the very latest version, please), Windows Media Player, and Sharepoint… Sharepoint! which nobody has; a veritable panoply of external dependencies each one of which is going to be a huge headache when you ship your application to a paying customer and it doesn’t work right. The technical name for this is DLL Hell. It works here: why doesn’t it work there?

The Raymond Chen Camp believes in making things easy for developers by making it easy to write once and run anywhere (well, on any Windows box). The MSDN Magazine Camp believes in making things easy for developers by giving them really powerful chunks of code which they can leverage, if they are willing to pay the price of incredibly complicated deployment and installation headaches, not to mention the huge learning curve. The Raymond Chen camp is all about consolidation. Please, don’t make things any worse, let’s just keep making what we already have still work. The MSDN Magazine Camp needs to keep churning out new gigantic pieces of technology that nobody can keep up with.

Here’s why this matters.

Microsoft Lost the Backwards Compatibility Religion

Inside Microsoft, the MSDN Magazine Camp has won the battle.

The first big win was making Visual Basic.NET not backwards-compatible with VB 6.0. This was literally the first time in living memory that when you bought an upgrade to a Microsoft product, your old data (i.e. the code you had written in VB6) could not be imported perfectly and silently. It was the first time a Microsoft upgrade did not respect the work that users did using the previous version of a product.

And the sky didn’t seem to fall, not inside Microsoft. VB6 developers were up in arms, but they were disappearing anyway, because most of them were corporate developers who were migrating to web development anyway. The real long term damage was hidden.

With this major victory under their belts, the MSDN Magazine Camp took over. Suddenly it was OK to change things. IIS 6.0 came out with a different threading model that broke some old applications. I was shocked to discover that our customers with Windows Server 2003 were having trouble running FogBugz. Then .NET 1.1 was not perfectly backwards compatible with 1.0. And now that the cat was out of the bag, the OS team got into the spirit and decided that instead of adding features to the Windows API, they were going to completely replace it. Instead of Win32, we are told, we should now start getting ready for WinFX: the next generation Windows API. All different. Based on .NET with managed code. XAML. Avalon. Yes, vastly superior to Win32, I admit it. But not an upgrade: a break with the past.

Outside developers, who were never particularly happy with the complexity of Windows development, have defected from the Microsoft platform en-masse and are now developing for the web. Paul Graham, who created Yahoo! Stores in the early days of the dotcom boom, summarized it eloquently: “There is all the more reason for startups to write Web-based software now, because writing desktop software has become a lot less fun. If you want to write desktop software now you do it on Microsoft’s terms, calling their APIs and working around their buggy OS. And if you manage to write something that takes off, you may find that you were merely doing market research for Microsoft.”

Microsoft got big enough, with too many developers, and they were too addicted to upgrade revenues, so they suddenly decided that reinventing everything was not too big a project. Heck, we can do it twice. The old Microsoft, the Microsoft of Raymond Chen, might have implemented things like Avalon, the new graphics system, as a series of DLLs that can run on any version of Windows and which could be bundled with applications that need them. There’s no technical reason not to do this. But Microsoft needs to give you a reason to buy Longhorn, and what they’re trying to pull off is a sea change, similar to the sea change that occurred when Windows replaced DOS. The trouble is that Longhorn is not a very big advance over Windows XP; not nearly as big as Windows was over DOS. It probably won’t be compelling enough to get people to buy all new computers and applications like they did for Windows. Well, maybe it will, Microsoft certainly needs it to be, but what I’ve seen so far is not very convincing. A lot of the bets Microsoft made are the wrong ones. For example, WinFS, advertised as a way to make searching work by making the file system be a relational database, ignores the fact that the real way to make searching work is by making searching work. Don’t make me type metadata for all my files that I can search using a query language. Just do me a favor and search the damned hard drive, quickly, for the string I typed, using full-text indexes and other technologies that were boring in 1973.

Automatic Transmissions Win the Day

Don’t get me wrong… I think .NET is a great development environment and Avalon with XAML is a tremendous advance over the old way of writing GUI apps for Windows. The biggest advantage of .NET is the fact that it has automatic memory management.

A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. It can be with reference counting or garbage collection; it can be Java, Lisp, Visual Basic (even 1.0), Smalltalk, or any of a number of scripting languages. If your programming language allows you to grab a chunk of memory without thinking about how it’s going to be released when you’re done with it, you’re using a managed-memory language, and you are going to be much more efficient than someone using a language in which you have to explicitly manage memory. Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it.

Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction. 2) Because you don’t have to spend any time writing code to free memory or tracking down memory leaks. 3) Because you don’t have to carefully coordinate the exit points from your functions to make sure things are cleaned up properly.

Racing car aficionados will probably send me hate mail for this, but my experience has been that there is only one case, in normal driving, where a good automatic transmission is inferior to a manual transmission. Similarly in software development: in almost every case, automatic memory management is superior to manual memory management and results in far greater programmer productivity.

If you were developing desktop applications in the early years of Windows, Microsoft offered you two ways to do it: writing C code which calls the Windows API directly and managing your own memory, or using Visual Basic and getting your memory managed for you. These are the two development environments I have used the most, personally, over the last 13 years or so, and I know them inside-out, and my experience has been that Visual Basic is significantly more productive. Often I’ve written the same code, once in C++ calling the Windows API and once in Visual Basic, and C++ always took three or four times as much work. Why? Memory management. The easiest way to see why is to look at the documentation for any Windows API function that needs to return a string. Look closely at how much discussion there is around the concept of who allocates the memory for the string, and how you negotiate how much memory will be needed. Typically, you have to call the function twice—on the first call, you tell it that you’ve allocated zero bytes, and it fails with a “not enough memory allocated” message and conveniently also tells you how much memory you need to allocate. That’s if you’re lucky enough not to be calling a function which returns a list of strings or a whole variable-length structure. In any case, simple operations like opening a file, writing a string, and closing it using the raw Windows API can take a page of code. In Visual Basic similar operations can take three lines.

So, you’ve got these two programming worlds. Everyone has pretty much decided that the world of managed code is far superior to the world of unmanaged code. Visual Basic was (and probably remains) the number one bestselling language product of all time and developers preferred it over C or C++ for Windows development, although the fact that “Basic” was in the name of the product made hardcore programmers shun it even though it was a fairly modern language with a handful of object-oriented features and very little leftover gunk (line numbers and the LET statement having gone the way of the hula hoop). The other problem with VB was that deployment required shipping a VB runtime, which was a big deal for shareware distributed over modems, and, worse, let other programmers see that your application was developed in (the shame!) Visual Basic.

One Runtime To Rule Them All

And along came .NET. This was a grand project, the super-duper unifying project to clean up the whole mess once and for all. It would have memory management, of course. It would still have Visual Basic, but it would gain a new language, one which is in spirit virtually the same as Visual Basic but with the C-like syntax of curly braces and semicolons. And best of all, the new Visual Basic/C hybrid would be called Visual C#, so you would not have to tell anyone you were a “Basic” programmer any more. All those horrid Windows functions with their tails and hooks and backwards-compatibility bugs and impossible-to-figure-out string-returning semantics would be wiped out, replaced by a single clean object oriented interface that only has one kind of string. One runtime to rule them all. It was beautiful. And they pulled it off, technically. .NET is a great programming environment that manages your memory and has a rich, complete, and consistent interface to the operating system and a rich, super complete, and elegant object library for basic operations.

And yet, people aren’t really using .NET much.

Oh sure, some of them are.

But the idea of unifying the mess of Visual Basic and Windows API programming by creating a completely new, ground-up programming environment with not one, not two, but three languages (or are there four?) is sort of like the idea of getting two quarreling kids to stop arguing by shouting “shut up!” louder than either of them. It only works on TV. In real life when you shout “shut up!” to two people arguing loudly you just create a louder three-way argument.

(By the way, for those of you who follow the arcane but politically-charged world of blog syndication feed formats, you can see the same thing happening over there. RSS became fragmented with several different versions, inaccurate specs and lots of political fighting, and the attempt to clean everything up by creating yet another format called Atom has resulted in several different versions of RSS plus one version of Atom, inaccurate specs and lots of political fighting. When you try to unify two opposing forces by creating a third alternative, you just end up with three opposing forces. You haven’t unified anything and you haven’t really fixed anything.)

So now instead of .NET unifying and simplifying, we have a big 6-way mess, with everybody trying to figure out which development strategy to use and whether they can afford to port their existing applications to .NET.

No matter how consistent Microsoft is in their marketing message (“just use .NET—trust us!”), most of their customers are still using C, C++, Visual Basic 6.0, and classic ASP, not to mention all the other development tools from other companies. And the ones that are using .NET are using ASP.NET to develop web applications, which run on a Windows server but don’t require Windows clients, which is a key point I’ll talk about more when I talk about the web.

Oh, Wait, There’s More Coming!

Now Microsoft has so many developers cranking away that it’s not enough to reinvent the entire Windows API: they have to reinvent it twice. At last year’s PDC they preannounced the next major version of their operating system, codenamed Longhorn, which will contain, among other things, a completely new user interface API, codenamed Avalon, rebuilt from the ground up to take advantage of modern computers’ fast display adapters and realtime 3D rendering. And if you’re developing a Windows GUI app today using Microsoft’s “official” latest-and-greatest Windows programming environment, WinForms, you’re going to have to start over again in two years to support Longhorn and Avalon. Which explains why WinForms is completely stillborn. Hope you haven’t invested too much in it. Jon Udell found a slide from Microsoft labelled “How Do I Pick Between Windows Forms and Avalon?” and asks, “Why do I have to pick between Windows Forms and Avalon?” A good question, and one to which he finds no great answer.

So you’ve got the Windows API, you’ve got VB, and now you’ve got .NET, in several language flavors, and don’t get too attached to any of that, because we’re making Avalon, you see, which will only run on the newest Microsoft operating system, which nobody will have for a loooong time. And personally I still haven’t had time to learn .NET very deeply, and we haven’t ported Fog Creek’s two applications from classic ASP and Visual Basic 6.0 to .NET because there’s no return on investment for us. None. It’s just Fire and Motion as far as I’m concerned: Microsoft would love for me to stop adding new features to our bug tracking software and content management software and instead waste a few months porting it to another programming environment, something which will not benefit a single customer and therefore will not gain us one additional sale, and therefore which is a complete waste of several months, which is great for Microsoft, because they have content management software and bug tracking software, too, so they’d like nothing better than for me to waste time spinning cycles catching up with the flavor du jour, and then waste another year or two doing an Avalon version, too, while they add features to their own competitive software. Riiiight.

No developer with a day job has time to keep up with all the new development tools coming out of Redmond, if only because there are too many dang employees at Microsoft making development tools!

It’s Not 1990

Microsoft grew up during the 1980s and 1990s, when the growth in personal computers was so dramatic that every year there were more new computers sold than the entire installed base. That meant that if you made a product that only worked on new computers, within a year or two it could take over the world even if nobody switched to your product. That was one of the reasons Word and Excel displaced WordPerfect and Lotus so thoroughly: Microsoft just waited for the next big wave of hardware upgrades and sold Windows, Word and Excel to corporations buying their next round of desktop computers (in some cases their first round). So in many ways Microsoft never needed to learn how to get an installed base to switch from product N to product N+1. When people get new computers, they’re happy to get all the latest Microsoft stuff on the new computer, but they’re far less likely to upgrade. This didn’t matter when the PC industry was growing like wildfire, but now that the world is saturated with PCs most of which are Just Fine, Thank You, Microsoft is suddenly realizing that it takes much longer for the latest thing to get out there. When they tried to “End Of Life” Windows 98, it turned out there were still so many people using it they had to promise to support that old creaking grandma for a few more years.

Unfortunately, these Brave New Strategies, things like .NET and Longhorn and Avalon, trying to create a new API to lock people into, can’t work very well if everybody is still using their good-enough computers from 1998. Even if Longhorn ships when it’s supposed to, in 2006, which I don’t believe for a minute, it will take a couple of years before enough people have it that it’s even worth considering as a development platform. Developers, developers, developers, and developers are not buying into Microsoft’s multiple-personality-disordered suggestions for how we should develop software.

Enter the Web

I’m not sure how I managed to get this far without mentioning the Web. Every developer has a choice to make when they plan a new software application: they can build it for the web or they can build a “rich client” application that runs on PCs. The basic pros and cons are simple: Web applications are easier to deploy, while rich clients offer faster response time enabling much more interesting user interfaces.

Web Applications are easier to deploy because there’s no installation involved. Installing a web application means typing a URL in the address bar. Today I installed Google’s new email application by typing Alt+D, gmail, Ctrl+Enter. There are far fewer compatibility problems and problems coexisting with other software. Every user of your product is using the same version so you never have to support a mix of old versions. You can use any programming environment you want because you only have to get it up and running on your own server. Your application is automatically available at virtually every reasonable computer on the planet. Your customers’ data, too, is automatically available at virtually every reasonable computer on the planet.

But there’s a price to pay in the smoothness of the user interface. Here are a few examples of things you can’t really do well in a web application:

  1. Create a fast drawing program
  2. Build a real-time spell checker with wavy red underlines
  3. Warn users that they are going to lose their work if they hit the close box of the browser
  4. Update a small part of the display based on a change that the user makes without a full roundtrip to the server
  5. Create a fast keyboard-driven interface that doesn’t require the mouse
  6. Let people continue working when they are not connected to the Internet

These are not all big issues. Some of them will be solved very soon by witty Javascript developers. Two new web applications, Gmail and Oddpost, both email apps, do a really decent job of working around or completely solving some of these issues. And users don’t seem to care about the little UI glitches and slowness of web interfaces. Almost all the normal people I know are perfectly happy with web-based email, for some reason, no matter how much I try to convince them that the rich client is, uh, richer.

So the Web user interface is about 80% there, and even without new web browsers we can probably get 95% there. This is Good Enough for most people and it’s certainly good enough for developers, who have voted to develop almost every significant new application as a web application.

Which means, suddenly, Microsoft’s API doesn’t matter so much. Web applications don’t require Windows.

It’s not that Microsoft didn’t notice this was happening. Of course they did, and when the implications became clear, they slammed on the brakes. Promising new technologies like HTAs and DHTML were stopped in their tracks. The Internet Explorer team seems to have disappeared; they have been completely missing in action for several years. There’s no way Microsoft is going to allow DHTML to get any better than it already is: it’s just too dangerous to their core business, the rich client. The big meme at Microsoft these days is: “Microsoft is betting the company on the rich client.” You’ll see that somewhere in every slide presentation about Longhorn. Joe Beda, from the Avalon team, says that “Avalon, and Longhorn in general, is Microsoft’s stake in the ground, saying that we believe power on your desktop, locally sitting there doing cool stuff, is here to stay. We’re investing on the desktop, we think it’s a good place to be, and we hope we’re going to start a wave of excitement…”

The trouble is: it’s too late.

I’m a Little Bit Sad About This, Myself

I’m actually a little bit sad about this, myself. To me the Web is great but Web-based applications with their sucky, high-latency, inconsistent user interfaces are a huge step backwards in daily usability. I love my rich client applications and would go nuts if I had to use web versions of the applications I use daily: Visual Studio, CityDesk, Outlook, Corel PhotoPaint, QuickBooks. But that’s what developers are going to give us. Nobody (by which, again, I mean “fewer than 10,000,000 people”) wants to develop for the Windows API any more. Venture Capitalists won’t invest in Windows applications because they’re so afraid of competition from Microsoft. And most users don’t seem to care about crappy Web UIs as much as I do.

And here’s the clincher: I noticed (and confirmed this with a recruiter friend) that Windows API programmers here in New York City who know C++ and COM programming earn about $130,000 a year, while typical Web programmers using managed code languages (Java, PHP, Perl, even ASP.NET) earn about $80,000 a year. That’s a huge difference, and when I talked to some friends from Microsoft Consulting Services about this they admitted that Microsoft had lost a whole generation of developers. The reason it takes $130,000 to hire someone with COM experience is because nobody bothered learning COM programming in the last eight years or so, so you have to find somebody really senior, usually they’re already in management, and convince them to take a job as a grunt programmer, dealing with (God help me) marshalling and monikers and apartment threading and aggregates and tearoffs and a million other things that, basically, only Don Box ever understood, and even Don Box can’t bear to look at them any more.

Much as I hate to say it, a huge chunk of developers have long since moved to the web and refuse to move back. Most .NET developers are ASP.NET developers, developing for Microsoft’s web server. ASP.NET is brilliant; I’ve been working with web development for ten years and it’s really just a generation ahead of everything out there. But it’s a server technology, so clients can use any kind of desktop they want. And it runs pretty well under Linux using Mono.

None of this bodes well for Microsoft and the profits it enjoyed thanks to its API power. The new API is HTML, and the new winners in the application development marketplace will be the people who can make HTML sing.

Rick Chapman is In Search of Stupidity

(This is the foreword to Rick Chapman’s new book, In Search of Stupidity.)

In every high tech company I’ve known, there’s a war going on, between the geeks and the suits.

Before you start reading this great new book full of propaganda from software marketing wizard and über-suit Rick Chapman, let me take a moment to tell you what the geeks think.

Play along with me for a minute, will you?

Please imagine the most stereotypically pale, Jolt-drinking, Chinese-food-eating, video-game-playing, slashdot-reading Linux-command-line-dwelling dork. Since this is just a stereotype, you should be free to imagine either a runt or a kind of chubby fellow, but in either case this is not the kind of person who plays football with his high school pals when he visits mom for Thanksgiving. Also, since he’s a stereotype, I shall not have to make complicated excuses for making him a him.

This is what our stereotypical programmer thinks: “Microsoft makes inferior products, but they have superior marketing, so everybody buys their stuff.”

Ask him what he thinks about the marketing people in his own company. “They’re really stupid. Yesterday I got into a big argument with this stupid sales chick in the break room and after ten minutes it was totally clear that she had no clue what the difference between 802.11a and 802.11b is. Duh!”

What do marketing people do, young geek? “I don’t know. They play golf with customers or something, when they’re not making me correct their idiot spec sheets. If it was up to me I’d fire ‘em all.”

A nice fellow named Jeffrey Tarter used to publish an annual list of the hundred largest personal computer software publishers called the Soft-letter 100. Here’s what the top ten looked like in 1984[1]:

Rank Company Annual Revenues
#1 Micropro International $60,000,000
#2 Microsoft Corp. $55,000,000
#3 Lotus $53,000,000
#4 Digital Research $45,000,000
#5 VisiCorp $43,000,000
#6 Ashton-Tate $35,000,000
#7 Peachtree $21,700,000
#8 MicroFocus $15,000,000
#9 Software Publishing $14,000,000
#10 Broderbund $13,000,000

OK, Microsoft is number 2, but it is one of a handful of companies with roughly similar annual revenues.

Now let’s look at the same list for 2001.

Rank Company Annual Revenues
#1 Microsoft Corp. $23,845,000,000
#2 Adobe $1,266,378,000
#3 Novell $1,103,592,000
#4 Intuit $1,076,000,000
#5 Autodesk $926,324,000
#6 Symantec $790,153,000
#7 Network Associates $745,692,000
#8 Citrix $479,446,000
#9 Macromedia $295,997,000
#10 Great Plains $250,231,000

Whoa. Notice, if you will, that every single company except Microsoft has disappeared from the top ten. Also notice, please, that Microsoft is so much larger than the next largest player, it’s not even funny. Adobe would double in revenues if they could just get Microsoft’s soda pop budget.

The personal computer software market is Microsoft. Microsoft’s revenues, it turns out, make up 69% of the total revenues of all the top 100 companies combined.

This is what we’re talking about, here.

Is this just superior marketing, as our imaginary geek claims? Or the result of an illegal monopoly? (Which begs the question: how did Microsoft get that monopoly? You can’t have it both ways.)

According to Rick Chapman, the answer is simpler: Microsoft was the only company on the list that never made a fatal, stupid mistake. Whether this was by dint of superior brainpower or just dumb luck, the biggest mistake Microsoft made was the dancing paperclip. And how bad was that, really? We ridiculed them, shut it off, and went back to using Word, Excel, Outlook, and Internet Explorer every minute of every day. But for every other software company that once had market leadership and saw it go down the drain, you can point to one or two giant blunders that steered the boat into an iceberg. Micropro fiddled around rewriting the printer architecture instead of upgrading their flagship product, WordStar. Lotus wasted a year and a half shoehorning 123 to run on 640kb machines; by the time they were done Excel was shipping and 640kb machines were a dim memory. Digital Research wildly overcharged for CP/M-86 and lost a chance to be the de-facto standard for PC operating systems. VisiCorp sued themselves out of existence. Ashton-Tate never missed an opportunity to piss off dBase developers, poisoning the fragile ecology that is so vital to a platform vendor’s success.

I’m a programmer, of  course, so I tend to blame the marketing people for these stupid mistakes. Almost all of them revolve around a failure of non-technical business people to understand basic technology facts. When Pepsi-pusher John Sculley was developing the Apple Newton, he didn’t know something that every computer science major in the country knows: handwriting recognition is not possible. This was at the same time that Bill Gates was hauling programmers into meetings begging them to create a single rich text edit control that could be reused in all their products. Put Jim Manzi (the suit who let the MBAs take over Lotus) in that meeting and he would be staring blankly. “What’s a rich text edit control?” It never would have occurred to him to take technological leadership because he didn’t grok the technology; in fact, the very use of the word grok in that sentence would probably throw him off.

In Search of Stupidity -- the Cover
Buy the book

If you ask me, and I’m biased, no software company can succeed unless there is a programmer at the helm. So far the evidence backs me up. But many of these boneheaded mistakes come from the programmers themselves. Netscape’s monumental decision to rewrite their browser instead of improving the old code base cost them several years of Internet time, during which their market share went from around 90% to about 4%, and this was the programmers’ idea. Of course, the nontechnical and inexperienced management of that company had no idea why this was a bad idea. There are still scads of programmers who defend Netscape’s ground-up rewrite. “The old code really sucked, Joel!” Yeah, uh-huh. Such programmers should be admired for their love of clean code, but they shouldn’t be allowed within 100 feet of any business decisions, since it’s obvious that clean code is more important to them than shipping, uh, software.

So I’ll concede to Rick a bit and say that if you want to be successful in the software business, you have to have a management team that thoroughly understands and loves programming, but they have to understand and love business, too. Finding a leader with strong aptitude in both dimensions is difficult, but it’s the only way to avoid making one of those fatal mistakes that Rick catalogs lovingly in this book. So read it, chuckle a bit, and if there’s a stupidhead running your company, get your résumé in shape and start looking for a house in Redmond.

[1] Source: Soft*letter, Jeffrey Tarter ed., April 30, 2001, 17:11.


Most software developers, including Fog Creek Software, are perfectly happy just to write software applications. You know, programs that do something or solve some particular problem. But the brave among us want to change the world more significantly, and they choose to work on platforms: big giant slabs of software that don’t quite do anything out of the box, but which enable a whole world of new and interesting applications. So they write operating systems, or DBMSs, or language runtimes like Java, and they hope to attract independent software developers to create the great new applications that their platforms enable.

Almost by definition, an operating system is a platform. Many platforms live on top of operating systems, such as the Java Runtime. And don’t forget that Windows didn’t start out as an OS, it started out as a program you ran on DOS which (out of the box) didn’t do much of anything, but enabled software developers to create GUI applications for inexpensive Intel boxes.

It’s really, really important to figure out if your product is a platform or not, because platforms need to be marketed in a very different way to be successful. That’s because a platform needs to appeal to developers first and foremost, not end users.

I just had the good fortune to read an advance copy of Rick Chapman’s excellent new book on stupidity in the software industry. (Prediction: best seller). Being an analytical kind of guy, I look for common themes. One of the biggest themes in software industry failures is a platform vendor that didn’t understand that they were a platform vendor, so they alienated their key constituency: the developers.

Example: NetWare took so long to release reasonable tools for creating NLMs that when Unix and Windows NT came along with superior and cheaper development tools, they swept away the mindshare of server software developers.

Example: Apple has spent decades making life miserable for their developers. Every new OS for almost 20 years required tweaks and changes to application code. If you got too successful, Apple competed against you (although sometimes they had a hand puppet called Claris compete against you so they could pretend it wasn’t them.)

Example: Developing software for OS/2 1.0 required an investment of $3000 for the SDK, and you had to write all your own printer drivers if printing was important for you. Printing was important, so OS/2 had no applications.

But the counter-examples are just as interesting:

Example: The first versions of Windows included a freely redistributable runtime so that if you wrote a Windows application, you could sell it to anyone with DOS, you weren’t limited to the few weirdo dorks (me!) who bought Windows 1.0.

Example: Despite the mistakes that Sun made with Java, the runtime was always free and good Java tools were cheap or free, too. No other development platform became so predominant so quickly (even Visual Basic, the top selling computer language of all time, took years to ramp up.)

If you want a platform to be successful, you need massive adoption, and that means you need developers to develop for it. The best way to kill a platform is to make it hard for developers to build on it. Most of the time, this happens because platform companies either don’t know that they have a platform (they think it’s an application) or they get greedy (they want all the revenue for themselves.)

Greedy platform companies can’t stand the idea of all kinds of unwashed riffraff making money off of their platform, so they make it darn near impossible for anyone to develop for it. Probably the most spectacular failure illustrating this was IBM’s PS/2, with its huge portfolio of proprietary technologies, such as the new Microchannel architecture designed to insure that only IBM could make expansion cards. This is, of course, monumentally shortsighted. Nobody wanted PS/2s because, uh, add-in cards weren’t available and they were too expensive when they were. As a platform vendor, you’re only as successful as the people who build on you.

A more subtle problem is when platform vendors don’t think they have a platform, they think they have an application. To illustrate this I’m once again going to have to pick on Groove.

“Why do you keep picking on Groove, Joel?” Three reasons:

  • They have an interesting architecture that provides important platform functionality which I could really use in my own products,
  • They make it impossible (or at least unrealistic) for ISVs to build on their platform, thus dooming themselves to oblivion, either out of greed or because they think Groove is an application, not a platform,
  • and Groove inventor Ray Ozzie has a weblog, so he can answer me if he thinks I’m off the mark. (He did.)

Here’s how I noticed the Groove problem. I’ve got a product, CityDesk, for simple desktop web content management. Weblogs, company sites, small organizations, etc. — people who need content management but can’t afford the big systems, don’t control a server anywhere, or just can’t be bothered to mess with installing perl scripts on a server.

As a 1.0 product we’ve got our share of weaknesses. One of the big ones is that people want to collaborate on CityDesk sites over the Internet. A reasonable request. For the next major release, we have to do something about this limitation. Basically, we’ve got two choices. The traditional choice would be to do something client-server: make a CityDesk server you can install somewhere so that everyone can collaborate.

But another choice, which would maintain CityDesk’s benefit of not requiring anything on the server, would be to use a secure peer-to-peer architecture. Something exactly like what Groove provides.

So I thought of porting CityDesk to Groove. Then I noticed that

  1. there is no free Groove runtime. Every single one of my customers would have to buy Groove.
  2. nobody has Groove yet.

That doomed the Groove idea from the start. I talked to some of the Groove “partners” who allegedly are developing software for Groove. “Was the Groove relationship worth anything?” I asked. “HA!” they said. “We paid $1500 and in exchange we get less than 10 clickthroughs a month from their web page. A waste of money. We couldn’t even get Groove to share their customer lists with us.”

This is not the kind of platform I want to develop for. Yet, technologically, it is exactly the kind of platform I want to develop for, but it’s controlled by a greedy (or clueless) company that is going to choke off their own oxygen — the compelling applications on top of Groove. Ray Ozzie is going nuts about how cool weblogs are — where’s the weblog application for Groove? Who’s going to write one? Evan Williams, creator of Blogger? Even Blogger Pro is only $35 a year and that doesn’t go very far to paying for a $99 Groove single user license.

What would happen if the Groove runtime were free? Follow the arc of Windows. It started out with a free runtime that let you run one GUI application at a time. Eventually, many people bought the full version to get the benefits of Windows file management, cut and paste between Windows applications, etc. Then Windows 3.0 came out and was so popular and had so many applications that it was bundled with every PC. Today Windows is like the television tax in Britain. Everybody pays it except the developer — when you’re writing software for Windows, it doesn’t cost one extra cent. In fact at no time in the history of Windows did developers ever have to worry about the cost of Windows itself.

Anybody who ever tried to sell software components (ActiveX controls, beans, etc.) knows that you have to have a royalty-free runtime or no developer will touch you with a 10 foot pole. Microsoft even lets you redistribute Jet, a complete relational database engine that is 9/10ths of Microsoft Access, for free. Heck, it’s preinstalled on Windows 2000.

If Groove wants to be a successful platform, they need to do the same thing. A free Groove redistributable would mean that hundreds of applications would spring up which would get the runtime spread far and wide. Many of those users would see the value in buying a full version of Groove with built in collaboration features. Groove services like the hosted reflectors would have a much larger potential audience.

Of course, they can continue down the path of Notes, assuming that the only way to sell software is to wow CEOs with cool Powerpoints and sell $1,000,000 corporate shelfware licenses. Eventually, this made some real money for Lotus, because Notes had one compelling application — email — built in. But imagine if the Notes runtime had been free. If Notes had a software industry sitting on top of it way back in the 80s, some dormroom startup might have made a compelling hypertext application for it instead and preempted the Web. The dreams of huge public Notes networks might have come true. Notes would be as common on PCs as Solitaire. Today it’s just Another Email System, one without much of a future.

I keep picking on Groove, but only because there’s something interesting in there, one of the few technologies that’s interesting enough to care about. Yes, the Groove engineers are architecture astronauts. That’s OK. They’re building architecture. But they’re positioning it like an application and I don’t think Groove will be successful if they do that. Someone else will come along with a P2P architecture and sell it like a component, or make it an open source library (yes, I’m aware of JXTA), and that’s what software developers will use.

Strategy Letter V

When I was in college I took two intro economics courses: macroeconomics and microeconomics. Macro was full of theories like “low unemployment causes inflation” that never quite stood up to reality. But the micro stuff was both cool and useful. It was full of interesting concepts about the relationships between supply and demand that really did work. For example, if you have a competitor who lowers their prices, the demand for your product will go down unless you match them.

In today’s episode, I’ll show how one of those concepts explains a lot about some familiar computer companies. Along the way, I noticed something interesting about open source software, which is this: most of the companies spending big money to develop open source software are doing it because it’s a good business strategy for them, not because they suddenly stopped believing in capitalism and fell in love with freedom-as-in-speech.

Every product in the marketplace has substitutes and complements. A substitute is another product you might buy if the first product is too expensive. Chicken is a substitute for beef. If you’re a chicken farmer and the price of beef goes up, the people will want more chicken, and you will sell more.

A complement is a product that you usually buy together with another product. Gas and cars are complements. Computer hardware is a classic complement of computer operating systems. And babysitters are a complement of dinner at fine restaurants. In a small town, when the local five star restaurant has a two-for-one Valentine’s day special, the local babysitters double their rates. (Actually, the nine-year-olds get roped into early service.)

All else being equal, demand for a product increases when the prices of its complements decrease.

Let me repeat that because you might have dozed off, and it’s important. Demand for a product increases when the prices of its complements decrease. For example, if flights to Miami become cheaper, demand for hotel rooms in Miami goes up — because more people are flying to Miami and need a room. When computers become cheaper, more people buy them, and they all need operating systems, so demand for operating systems goes up, which means the price of operating systems can go up.

At this point, it’s pretty common for people to try to confuse things by saying, “aha! But Linux is FREE!” OK. First of all, when an economist considers price, they consider the total price, including some intangible things like the time it takes to set up, reeducate everyone, and convert existing processes. All the things that we like to call “total cost of ownership.”

Secondly, by using the free-as-in-beer argument, these advocates try to believe that they are not subject to the rules of economics because they’ve got a nice zero they can multiply everything by. Here’s an example. When Slashdot asked Linux developer Moshe Bar if future Linux kernels would be compatible with existing device drivers, he said that they didn’t need to. “Proprietary software goes at the tariff of US$ 50-200 per line of debugged code. No such price applies to OpenSource software.” Moshe goes on to claim that it’s OK for every Linux kernel revision to make all existing drivers obsolete, because the cost of rewriting all those existing drivers is zero. This is completely wrong. He’s basically claiming that spending a small amount of programming time making the kernel backwards compatible is equivalent to spending a huge amount of programming time rewriting every driver, because both numbers are multiplied by their “cost,” which he believes to be zero. This is a prima facie fallacy. The thousands or millions of developer hours it takes to revise every existing device driver are going to have to come at the expense of something. And until that’s done, Linux will be once again handicapped in the marketplace because it doesn’t support existing hardware. Wouldn’t it be better to use all that “zero cost” effort making Gnome better? Or supporting new hardware?

Debugged code is NOT free, whether proprietary or open source. Even if you don’t pay cash dollars for it, it has opportunity cost, and it has time cost. There is a finite amount of volunteer programming talent available for open source work, and each open source project competes with each other open source project for the same limited programming resource, and only the sexiest projects really have more volunteer developers than they can use. To summarize, I’m not very impressed by people who try to prove wild economic things about free-as-in-beer software, because they’re just getting divide-by-zero errors as far as I’m concerned.

Eazel closed.Open source is not exempt from the laws of gravity or economics. We saw this with Eazel, ArsDigita, The Company Formerly Known as VA Linux and a lot of other attempts. But something is still going on which very few people in the open source world really understand: a lot of very large public companies, with responsibilities to maximize shareholder value, are investing a lot of money in supporting open source software, usually by paying large teams of programmers to work on it. And that’s what the principle of complements explains.

Once again: demand for a product increases when the price of its complements decreases. In general, a company’s strategic interest is going to be to get the price of their complements as low as possible. The lowest theoretically sustainable price would be the “commodity price” — the price that arises when you have a bunch of competitors offering indistinguishable goods. So:

Smart companies try to commoditize their products’ complements.

If you can do this, demand for your product will increase and you will be able to charge more and make more.

When IBM designed the PC architecture, they used off-the-shelf parts instead of custom parts, and they carefully documented the interfaces between the parts in the (revolutionary) IBM-PC Technical Reference Manual. Why? So that other manufacturers could join the party. As long as you match the interface, you can be used in PCs. IBM’s goal was to commoditize the add-in market, which is a complement of the PC market, and they did this quite successfully. Within a short time scrillions of companies sprung up offering memory cards, hard drives, graphics cards, printers, etc. Cheap add-ins meant more demand for PCs.

When IBM licensed the operating system PC-DOS from Microsoft, Microsoft was very careful not to sell an exclusive license. This made it possible for Microsoft to license the same thing to Compaq and the other hundreds of OEMs who had legally cloned the IBM PC using IBM’s own documentation. Microsoft’s goal was to commoditize the PC market. Very soon the PC itself was basically a commodity, with ever decreasing prices, consistently increasing power, and fierce margins that make it extremely hard to make a profit. The low prices, of course, increase demand. Increased demand for PCs meant increased demand for their complement, MS-DOS. All else being equal, the greater the demand for a product, the more money it makes for you. And that’s why Bill Gates can buy Sweden and you can’t.

This year Microsoft’s trying to do it again: their new game console, the XBox, uses commodity PC hardware instead of custom parts. The theory (explained in this book) was that commodity hardware gets cheaper every year, so the XBox could ride down the prices. Unfortunately it seems to have backfired: apparently commodity PC hardware has already been squeezed down to commodity prices, and so the price of making an XBox isn’t declining as fast as Microsoft would like. The other part of Microsoft’s XBox strategy was to use DirectX, a graphics library that can be used to write code that runs on all kinds of video chips. The goal here is to make the video chip a commodity, to lower its price, so that more games are sold, where the real profits occur. And why don’t the video chip vendors of the world try to commoditize the games, somehow? That’ s a lot harder. If the game Halo is selling like crazy, it doesn’t really have any substitutes. You’re not going to go to the movie theatre to see Star Wars: Attack of the Clones and decide instead that you would be satisfied with a Woody Allen movie. They may both be great movies, but they’re not perfect substitutes. Now: who would you rather be, a game publisher or a video chip vendor?

Commoditize your complements.

Understanding this strategy actually goes a long, long way in explaining why many commercial companies are making big contributions to open source. Let’s go over these.

Headline: IBM Spends Millions to Develop Open Source Software.

Myth: They’re doing this because Lou Gerstner read the GNU Manifesto and decided he doesn’t actually like capitalism.

Reality: They’re doing this because IBM is becoming an IT consulting company. IT consulting is a complement of enterprise software. Thus IBM needs to commoditize enterprise software, and the best way to do this is by supporting open source. Lo and behold, their consulting division is winning big with this strategy.

Headline: Netscape Open Sources Their Web Browser.

Myth: They’re doing this to get free source code contributions from people in cybercafes in New Zealand.

Reality: They’re doing this to commoditize the web browser.

This has been Netscape’s strategy from day one. Have a look at the very first Netscape press release: the browser is “freeware.” Netscape gave away the browser so they could make money on servers. Browsers and servers are classic complements. The cheaper the browsers, the more servers you sell. This was never as true as it was in October 1994. (Netscape was actually surprised when MCI came in the door and dumped so much money in their laps that they realized they could make money off of the browser, too. This wasn’t required by the business plan.)

When Netscape released Mozilla as Open Source, it was because they saw an opportunity to lower the cost of developing the browser. So they could get the commodity benefits at a lower cost.

Later AOL/Time Warner acquired Netscape. The server software, which was supposed to be the beneficiary of commodity browsers, wasn’t doing all that well, and was jettisoned. Now: why would AOL/Time Warner continue to invest anything in open source?

AOL/Time Warner is an entertainment company. Entertainment companies are the complement of entertainment delivery platforms of all types, including web browsers. This giant conglomerate’s strategic interest is to make entertainment delivery – web browsers – a commodity for which nobody can charge money.

My argument is a little bit tortured by the fact that Internet Explorer is free-as-in-beer. Microsoft wanted to make web browsers a commodity, too, so they can sell desktop and server operating systems. They went a step further and delivered a collection of components which anyone could use to throw together a web browser. Neoplanet, AOL, and Juno used these components to build their own web browsers. Given that IE is free, what is the incentive for Netscape to make the browser “even cheaper”? It’s a preemptive move. They need to prevent Microsoft getting a complete monopoly in web browsers, even free web browsers, because that would theoretically give Microsoft an opportunity to increase the cost of web browsing in other ways — say, by increasing the price of Windows.

(My argument is even more shaky because it’s pretty clear that Netscape in the days of Barksdale didn’t exactly know what it was doing. A more likely explanation for what Netscape did is that upper management was technologically inept, and they had no choice but to go along with whatever scheme the developers came up with. The developers were hackers, not economists, and only coincidentally came up with a scheme which serves their strategy. But let’s give them the benefit of the doubt.)

Headline: Transmeta Hires Linus, Pays Him To Hack on Linux.

Myth: They just did it to get publicity. Would you have heard of Transmeta otherwise?

Reality: Transmeta is a CPU company. The natural complement of a CPU is an operating system. Transmeta wants OSs to be a commodity.

Headline: Sun and HP Pay Ximian To Hack on Gnome.

Myth: Sun and HP are supporting free software because they like Bazaars, not Cathedrals.

Reality: Sun and HP are hardware companies. They make boxen. In order to make money on the desktop, they need for windowing systems, which are a complement of desktop computers, to be a commodity. Why don’t they take the money they’re paying Ximian and use it to develop a proprietary windowing system? They tried this (Sun had NeWS and HP had New Wave), but these are really hardware companies at heart with pretty crude software skills, and they need windowing systems to be a cheap commodity, not a proprietary advantage which they have to pay for. So they hired the nice guys at Ximian to do this for the same reason that Sun bought Star Office and open sourced it: to commoditize software and make more money on hardware.

Headline: Sun Develops Java; New “Bytecode” System Means Write Once, Run Anywhere.

The bytecode idea is not new — programmers have always tried to make their code run on as many machines as possible. (That’s how you commoditize your complement). For years Microsoft had its own p-code compiler and portable windowing layer which let Excel run on Mac, Windows, and OS/2, and on Motorola, Intel, Alpha, MIPS and PowerPC chips. Quark has a layer which runs Macintosh code on Windows. The C programming language is best described as a hardware-independent assembler language. It’s not a new idea to software developers.

If you can run your software anywhere, that makes hardware more of a commodity. As hardware prices go down, the market expands, driving more demand for software (and leaving customers with extra money to spend on software which can now be more expensive.)

Sun’s enthusiasm for WORA is, um, strange, because Sun is a hardware company. Making hardware a commodity is the last thing they want to do.


Sun is the loose cannon of the computer industry. Unable to see past their raging fear and loathing of Microsoft, they adopt strategies based on anger rather than self-interest. Sun’s two strategies are (a) make software a commodity by promoting and developing free software (Star Office, Linux, Apache, Gnome, etc), and (b) make hardware a commodity by promoting Java, with its bytecode architecture and WORA. OK, Sun, pop quiz: when the music stops, where are you going to sit down? Without proprietary advantages in hardware or software, you’re going to have to take the commodity price, which barely covers the cost of cheap factories in Guadalajara, not your cushy offices in Silicon Valley.

“But Joel!” Jared says. “Sun is trying to commoditize the operating system, like Transmeta, not the hardware.” Maybe, but the fact that Java bytecode also commoditizes the hardware is some pretty significant collateral damage to sustain.

An important thing you notice from all these examples is that it’s easy for software to commoditize hardware (you just write a little hardware abstraction layer, like Windows NT’s HAL, which is a tiny piece of code), but it’s incredibly hard for hardware to commoditize software. Software is not interchangable, as the StarOffice marketing team is learning. Even when the price is zero, the cost of switching from Microsoft Office is non-zero. Until the switching cost becomes zero, desktop office software is not truly a commodity. And even the smallest differences can make two software packages a pain to switch between. Despite the fact that Mozilla has all the features I want and I’d love to use it if only to avoid the whack-a-mole pop-up-ad game, I’m too used to hitting Alt+D to go to the address bar. So sue me. One tiny difference and you lose your commodity status. But I’ve pulled hard drives out of IBM computers and slammed them into Dell computers and, boom, the system comes up perfectly and runs as if it were still in the old computer.

Amos Michelson, the CEO of Creo, told me that every employee in his firm is required to take a course in what he calls “economic thinking.” Great idea. Even simple concepts in basic microeconomics go a long way to understanding some of the fundamental shifts going on today.


Fire And Motion

Sometimes I just can’t get anything done.

Sure, I come into the office, putter around, check my email every ten seconds, read the web, even do a few brainless tasks like paying the American Express bill. But getting back into the flow of writing code just doesn’t happen.

TetrisThese bouts of unproductiveness usually last for a day or two. But there have been times in my career as a developer when I went for weeks at a time without being able to get anything done. As they say, I’m not in flow. I’m not in the zone. I’m not anywhere.

Everybody has mood swings; for some people they are mild, for others, they can be more pronounced or even dysfunctional. And the unproductive periods do seem to correlate somewhat with gloomier moods.

It makes me think of those researchers who say that basically people can’t control what they eat, so any attempt to diet is bound to be short term and they will always yoyo back to their natural weight. Maybe as a software developer I really can’t control when I’m productive, and I just have to take the slow times with the fast times and hope that they average out to enough lines of code to make me employable.


Go read The Onion for a while.

What drives me crazy is that ever since my first job I’ve realized that as a developer, I usually average about two or three hours a day of productive coding. When I had a summer internship at Microsoft, a fellow intern told me he was actually only going into work from 12 to 5 every day. Five hours, minus lunch, and his team loved him because he still managed to get a lot more done than average. I’ve found the same thing to be true. I feel a little bit guilty when I see how hard everybody else seems to be working, and I get about two or three quality hours in a day, and still I’ve always been one of the most productive members of the team. That’s probably why when Peopleware and XP insist on eliminating overtime and working strictly 40 hour weeks, they do so secure in the knowledge that this won’t reduce a team’s output.

But it’s not the days when I “only” get two hours of work done that worry me. It’s the days when I can’t do anything.

I’ve thought about this a lot. I tried to remember the time when I got the most work done in my career. It was probably when Microsoft moved me into a beautiful, plush new office with large picture windows overlooking a pretty stone courtyard full of cherry trees in bloom. Everything was clicking. For months I worked nonstop grinding out the detailed specification for Excel Basic — a monumental ream of paper going into incredible detail covering a gigantic object model and programming environment. I literally never stopped. When I had to go to Boston for MacWorld I took a laptop with me, and documented the Window class sitting on a pleasant terrace at HBS.

Once you get into flow it’s not too hard to keep going. Many of my days go like this: (1) get into work (2) check email, read the web, etc. (3) decide that I might as well have lunch before getting to work (4) get back from lunch (5) check email, read the web, etc. (6) finally decide that I’ve got to get started (7) check email, read the web, etc. (8) decide again that I really have to get started (9) launch the damn editor and (10) write code nonstop until I don’t realize that it’s already 7:30 pm.

Somewhere between step 8 and step 9 there seems to be a bug, because I can’t always make it across that chasm.bike trip For me, just getting started is the only hard thing. An object at rest tends to remain at rest. There’s something incredible heavy in my brain that is extremely hard to get up to speed, but once it’s rolling at full speed, it takes no effort to keep it going. Like a bicycle decked out for a cross-country, self-supported bike trip — when you first start riding a bike with all that gear, it’s hard to believe how much work it takes to get rolling, but once you are rolling, it feels just as easy as riding a bike without any gear.

Maybe this is the key to productivity: just getting started. Maybe when pair programming works it works because when you schedule a pair programming session with your buddy, you force each other to get started.

Joel in the Army

When I was an Israeli paratrooper a general stopped by to give us a little speech about strategy. In infantry battles, he told us, there is only one strategy: Fire and Motion. You move towards the enemy while firing your weapon. The firing forces him to keep his head down so he can’t fire at you. (That’s what the soldiers mean when they shout “cover me.” It means, “fire at our enemy so he has to duck and can’t fire at me while I run across this street, here.” It works.)  The motion allows you to conquer territory and get closer to your enemy, where your shots are much more likely to hit their target. If you’re not moving, the enemy gets to decide what happens, which is not a good thing. If you’re not firing, the enemy will fire at you, pinning you down.

I remembered this for a long time. I noticed how almost every kind of military strategy, from air force dogfights to large scale naval maneuvers, is based on the idea of Fire and Motion. It took me another fifteen years to realize that the principle of Fire and Motion is how you get things done in life. You have to move forward a little bit, every day. It doesn’t matter if your code is lame and buggy and nobody wants it. If you are moving forward, writing code and fixing bugs constantly, time is on your side. Watch out when your competition fires at you. Do they just want to force you to keep busy reacting to their volleys, so you can’t move forward?

Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That’s probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP. The companies who stumble are the ones who spend too much time reading tea leaves to figure out the future direction of Microsoft. People get worried about .NET and decide to rewrite their whole architecture for .NET because they think they have to. Microsoft is shooting at you, and it’s just cover fire so that they can move forward and you can’t, because this is how the game is played, Bubby. Are you going to support Hailstorm? SOAP? RDF? Are you supporting it because your customers need it, or because someone is firing at you and you feel like you have to respond? The sales teams of the big companies understand cover fire. They go into their customers and say, “OK, you don’t have to buy from us. Buy from the best vendor. But make sure that you get a product that supports (XML / SOAP / CDE / J2EE) because otherwise you’ll be Locked In The Trunk.” Then when the little companies try to sell into that account, all they hear is obedient CTOs parrotting “Do you have J2EE?” And they have to waste all their time building in J2EE even if it doesn’t really make any sales, and gives them no opportunity to distinguish themselves. It’s a checkbox feature — you do it because you need the checkbox saying you have it, but nobody will use it or needs it. And it’s cover fire.

Fire and Motion, for small companies like mine, means two things. You have to have time on your side, and you have to move forward every day. Sooner or later you will win. All I managed to do yesterday is improve the color scheme in FogBUGZ just a little bit. That’s OK. It’s getting better all the time. Every day our software is better and better and we have more and more customers and that’s all that matters. Until we’re a company the size of Oracle, we don’t have to think about grand strategies. We just have to come in every morning and somehow, launch the editor.

It's getting better all the time... o/~