Posts by Joel Spolsky

In 2000 I co-founded Fog Creek Software, where we created lots of cool things like the FogBugz bug tracker, Trello, and Glitch. I also worked with Jeff Atwood to create Stack Overflow and served as CEO of Stack Overflow from 2010-2019. Today I serve as the chairman of the board for Stack Overflow, Glitch, and HASH.

2002/02/09

“I don’t know what’s wrong with my development team,” the CEO thinks to himself. “Things were going so well when we started this project. For the first couple of weeks, the team cranked like crazy and got a great prototype working. But since then, things seem to have slowed to a crawl. They’re just not working hard any more.” He chooses a Callaway Titanium Driver and sends the caddy to fetch an ice-cold lemonade. “Maybe if I fire a couple of laggards that’ll light a fire under them!”

The Iceberg Secret, Revealed

The Iceberg Secret, Revealed

“I don’t know what’s wrong with my development team,” the CEO thinks to himself. “Things were going so well when we started this project. For the first couple of weeks, the team cranked like crazy and got a great prototype working. But since then, things seem to have slowed to a crawl. They’re just not working hard any more.” He chooses a Callaway Titanium Driver and sends the caddy to fetch an ice-cold lemonade. “Maybe if I fire a couple of laggards that’ll light a fire under them!”

Meanwhile, of course, the development team has no idea that anything’s wrong. In fact, nothing is wrong. They’re right on schedule.

Don’t let this happen to you! I’m going to let you in on a little secret about those non-technical management types that will make your life a million times easier. It’s real simple. Once you know my secret, you’ll never have trouble working with non-technical managers again (unless you get into an argument over the coefficient of restitution of their golf clubs).

It’s pretty clear that programmers think in one language, and MBAs think in another. I’ve been thinking about the problem of communication in software management for a while, because it’s pretty clear to me that the power and rewards accrue to those rare individuals who know how to translate between Programmerese and MBAese.

[Image]

Since I started working in the software industry, almost all the software I’ve worked on has been what might be called “speculative” software. That is, the software is not being built for a particular customer — it’s being built in hopes that zillions of people will buy it. But many software developers don’t have that luxury. They may be consultants developing a project for a single client, or they may be in-house programmers working on a complicated corporate whatsit for Accounting (or whatever it is you in-house programmers do; it’s rather mysterious to me).

Have you ever noticed that on these custom projects, the single most common cause of overruns, failures, and general miserableness always boils down to, basically, “the (insert expletive here) customer didn’t know what they wanted?”

Here are three versions of the same pathology:

  1. “The damn customer kept changing his mind. First he wanted Client/Server. Then he read about XML in Delta Airlines Inflight Magazine and decided he had to have XML. Now we’re rewriting the thing to use fleets of small Lego Mindstorms Robots.”
  2. “We built it exactly the way they wanted. The contract specified the whole thing down to the smallest detail. We delivered exactly what the contract said. But when we delivered it, they were crestfallen.”
  3. “Our miserable sales person agreed to a fixed price contract to build what was basically unspecified, and the customer’s lawyers were sharp enough to get a clause in the contract that they don’t have to pay us until ‘acceptance by customer,’ so we had to put a team of nine developers on their project for two years and only got paid $800.”

If there’s one thing every junior consultant needs to have injected into their head with a heavy duty 2500 RPM DeWalt Drill, it’s this: Customers Don’t Know What They Want. Stop Expecting Customers to Know What They Want. It’s just never going to happen. Get over it.

Instead, assume that you’re going to have to build something anyway, and the customer is going to have to like it, but they’re going to be a little bit surprised. YOU have to do the research. YOU have to figure out a design that solves the problem that the customer has in a pleasing way.

Put yourself in their shoes. Imagine that you’ve just made $100,000,000 selling your company to Yahoo!, and you’ve decided that it’s about time to renovate your kitchen. So you hire an expert architect with instructions to make it “as cool as Will and Grace’s Kitchen.” You have no idea how to accomplish this. You don’t know that you want a Viking stove and a Subzero refrigerator — these are not words in your vocabulary. You want the architect to do something good, that’s why you hired him.

The Extreme Programming folks say that the solution to this is to get the customer in the room and involve them in the design process every step of the way, as a member of the development team. This is, I think, a bit too “extreme.” It’s as if my architect made me show up while they were designing the kitchen and asked me to provide input on every little detail. It’s boring for me, if I wanted to be an architect I would have become an architect.

Anyway, you don’t really want a customer on your team, do you? The customer-nominee is just as likely to wind up being some poor dweeb from Accounts Payable who got sent to work with the programmers because he was the slowest worker over there and they would barely notice his absence. And you’re just going to spend all your design time explaining things in words of one syllable.

Assume that your customers don’t know what they want. Design it yourself, based on your understanding of the domain. If you need to spend some time learning about the domain or if you need a domain expert to help you, that’s fine, but the design of the software is your job. If you do your domain homework and create a good UI, the customer will be pleased.

Now, I promised to tell you a secret about translating between the language of the customers (or nontechnical managers) of your software and the language of programmers.

You know how an iceberg is 90% underwater? Well, most software is like that too — there’s a pretty user interface that takes about 10% of the work, and then 90% of the programming work is under the covers. And if you take into account the fact that about half of your time is spent fixing bugs, the UI only takes 5% of the work. And if you limit yourself to the visual part of the UI, the pixels, what you would see in PowerPoint, now we’re talking less than 1%.

That’s not the secret. The secret is that People Who Aren’t Programmers Do Not Understand This.

There are some very, very important corollaries to the Iceberg Secret.

Important Corollary One. If you show a nonprogrammer a screen which has a user interface that is 90% worse, they will think that the program is 90% worse.

I learned this lesson as a consultant, when I did a demo of a major web-based project for a client’s executive team. The project was almost 100% code complete. We were still waiting for the graphic designer to choose fonts and colors and draw the cool 3-D tabs. In the meantime, we just used plain fonts and black and white, there was a bunch of ugly wasted space on the screen, basically it didn’t look very good at all. But 100% of the functionality was there and was doing some pretty amazing stuff.

What happened during the demo? The clients spent the entire meeting griping about the graphical appearance of the screen. They weren’t even talking about the UI. Just the graphical appearance. “It just doesn’t look slick,” complained their project manager. That’s all they could think about. We couldn’t get them to think about the actual functionality. Obviously fixing the graphic design took about one day. It was almost as if they thought they had hired painters.

Important Corollary Two. If you show a nonprogrammer a screen which has a user interface which is 100% beautiful, they will think the program is almost done.

People who aren’t programmers are just looking at the screen and seeing some pixels. And if the pixels look like they make up a program which does something, they think “oh, gosh, how much harder could it be to make it actually work?

The big risk here is that if you mock up the UI first, presumably so you can get some conversations going with the customer, then everybody’s going to think you’re almost done. And then when you spend the next year working “under the covers,” so to speak, nobody will really see what you’re doing and they’ll think it’s nothing.

Important Corollary Three. The dotcom that has the cool, polished looking web site and about four web pages will get a higher valuation than the highly functional dotcom with 3700 years of archives and a default grey background.

Oh, wait, dotcoms aren’t worth anything any more. Never mind.

Important Corollary Four. When politics demands that various nontechnical managers or customers “sign off” on a project, give them several versions of the graphic design to choose from.

Vary the placement of some things, change the look and feel and fonts, move the logo and make it bigger or smaller. Let them feel important by giving them non-crucial lipstick-on-a-chicken stuff to muck around with. They can’t do much damage to your schedule here. A good interior decorator is constantly bringing their client swatches and samples and stuff to choose from. But they would never discuss dishwasher placement with the client. It goes next to the sink, no matter what the client wants. There’s no sense wasting time arguing about where the dishwasher goes, it has to go next to the sink, don’t even bring it up; let the clients get their design kicks doing some harmless thing like changing their mind 200 times about whether to use Italian Granite or Mexican Tiles or Norwegian wood butcher-block for the countertops.

Important Corollary Five. When you’re showing off, the only thing that matters is the screen shot. Make it 100% beautiful.

Don’t, for a minute, think that you can get away with asking anybody to imagine how cool this would be. Don’t think that they’re looking at the functionality. They’re not. They want to see pretty pixels.

Steve Jobs understands this. Oh boy does he understand this. Engineers at Apple have learned to do things that make for great screen shots, like the gorgeous new 1024×1024 icons in the dock, even if they waste valuable real estate. And the Linux desktop crowd goes crazy about semitransparent xterms, which make for good screenshots but are usually annoying to use. Every time Gnome or KDE announces a new release I go straight to the screenshots and say, “oh, they changed the planet from Jupiter to Saturn. Cool.” Never mind what they really did.

Remember the CEO at the beginning of this article? He was unhappy because his team had showed him great PowerPoints at the beginning — mockups, created in Photoshop, not even VB. And now that they’re actually getting stuff done under the covers, it looks like they’re not doing anything.

What can you do about this? Once you understand the Iceberg Secret, it’s easy to work with it. Understand that any demos you do in a darkened room with a projector are going to be all about pixels. If you can, build your UI in such a way that unfinished parts look unfinished. For example, use scrawls for the icons on the toolbar until the functionality is there. As you’re building your web service, you may want to consider actually leaving out features from the home page until those features are built. That way people can watch the home page go from 3 commands to 20 commands as more things get built.

More importantly, make sure you control what people think about the schedule. Provide a detailed schedule in Excel format. Every week, send out self-congratulatory email talking about how you’ve moved from 32% complete to 35% complete and are on track to ship on December 25th. Make sure that the actual facts dominate any thinking about whether the project is moving forward at the right speed. And don’t let your boss use Callaway Titanium Drivers, I don’t care how much you want him to win, the USGA has banned them and it’s just not fair.

Discuss

2002/02/08

Now that ArsDigita is gone, I think we can officially declare the Internet Exuberance era, which opened with Travels with Samantha, officially closed. Even though the tendency is to blame it on the VCs, who replaced an exciting, charismatic and visionary founder with grey, grey, grey, the truth is that the Internet consulting market was totally wiped out more than a year ago and there’s no reason for AD to have been exempted. Fog Creek only survived because we started the company late enough in the game that we never had time to build much of a consulting business, and we never open sourced the crown jewels so we had something to sell.

By the way, have you noticed how often companies officially sell themselves off (probably for pennies) rather than simply liquidate? That’s just to make the executives’ bios look better. They can say on their resume, “I was the CxO of ArsDigita which was later sold to Red Hat” instead of saying “I managed to turn $40 million of VC and a profitable company into a heaping pile of mineral-encrusted fish tanks.” Red Hat is a part of the Greylock Keiretsu, too.

And even though nobody ever got the Ferrari, it sure made a big splash, didn’t it.

TV: Oops, they forgot to tell me that they actually wanted me to be in San Francisco for this interview. So it will be rescheduled. Whew!

2002/02/06

TechInterview: “I have a black triangle, a white triangle, a black circle and a white circle…”

These classic problems are reported in [Shafir 1994]. Read it if you can get your hands on it. The interesting thing for us as programmers is that certain types of logical thinking are inherently hard. In particular, most people will stop working through a problem when they get to a disjunction, that is, a point on the decision tree where you need more information. This becomes especially interesting when you don’t actually need the information because the result would be the same in any case. But there is something that makes most people unwilling to go past uncertainty points in problem solving anyway. Good programmers usually overcome this problem. It’s a critical skill in debugging — we don’t want to waste time investigating areas that could not possibly cause the bug. This might be one of those interesting brain-properties that distinguishes great programmers from everyone else.

Shafir, Eldar. (1994): Uncertainty and the difficulty of thinking through disjunctions.  Cognition, 50, 403-430.  (Reprinted in J. Mehler & S. Franck (Eds.), Cognition on Cognition, Cambridge, MA: MIT Press, 1995.)

2002/01/31

This is one of those days where I could write a comment that says:

maybe it’s easier to do this at eval time.
We could tag the foreach onto the malkovitch.

and it would actually make sense.

What is Six Degrees?

When software companies say that “their capital assets go home every night,” they are serious. Dead serious. And when Quark goes down in flames because the bean-counters pushed out the experienced developers and hired new, cheaper ones 10 time zones away, there will be a great case study in Harvard Business Review to prove it.

There are not a lot of software companies to work for in Denver. So when the key designers of QuarkXPress 5.0 got sick of being treated like interchangable code slaves, they started their own shop and quickly sold it to a smart Vancouver-based company that wrote them a blank check and let them stay in their hometown.

What they came up with is nothing short of a breakthrough in the field of personal knowledge management, an effervescent field that has not quite made it onto the radar of the buzzword maniacs. Six Degrees is gonna be Google for the desktop.  Everywhere I look I see people blabbing about how they’re going to crack the knowledge management nut. It’s like 7 years ago when everybody was making a search engine. But only Larry and Sergey figured out the formula to make search actually work, and only Mark, David, Cameron and Zach figured out the formula for to make knowledge management actually work. This is cool stuff. Ship it already, dammit, I need it yesterday!

2002/01/24

Rub-a-dub-dub

One reason people are tempted to rewrite their entire code base from scratch is that the original code base wasn’t designed for what it’s doing. It was designed as a prototype, an experiment, a learning exercise, a way to go from zero to IPO in nine months, or a one-off demo. And now it has grown into a big mess that’s fragile and impossible to add code to, and everybody’s whiny, and the old programmers quit in despair and the new ones that are brought in can’t make head or tail of the code so they somehow convince management to give up and start over while Microsoft takes over their business. Today let me tell you a story about what they could have done instead.

New Web Shop for Fog Creek Software

Fog Creek StoreWe got sick of paying the 13.9% transaction charge that Digibuy is charging us for every software download, and we needed more control over our commerce server so that we could run a proper affiliate program. So we got our own merchant account and rolled our own software for it. It’s been fun, and the transaction cost should be down to about 2.5%.

Patching Code?

I wanted to use .RTPatch for the upcoming CityDesk service pack, but it costs $5000 for one license. Yipes. That’s a lot to pay for what computer scientists call [Aho75]. Can anybody suggest an alternative?

Rub a dub dub

One reason people are tempted to rewrite their entire code base from scratch is that the original code base wasn’t designed for what it’s doing. It was designed as a prototype, an experiment, a learning exercise, a way to go from zero to IPO in nine months, or a one-off demo. And now it has grown into a big mess that’s fragile and impossible to add code to, and everybody’s whiny, and the old programmers quit in despair and the new ones that are brought in can’t make head or tail of the code so they somehow convince management to give up and start over while Microsoft takes over their business. Today let me tell you a story about what they could have done instead.

FogBUGZ started out six years ago as an experiment to teach myself ASP programming. Pretty soon it became an in-house bug tracking system. It got embellished almost daily with features that people needed until it was good enough that it no longer justified any more work.

Various friends asked me if they could use FogBUGZ at their companies. The trouble was, it had too many hardcoded things that made it a pain to run anywhere other than on the original computer where it was deployed. I had used a bunch of SQL Server stored procedures, which meant that you needed SQL Server to run FogBUGZ, which was expensive and overkill for some of the two person teams that wanted to run it. And so on. So I would tell my friends, “gosh, for $5000 in consulting fees, I’ll spend a couple of days and clean up the code so you can run it on your server using Access instead of SQL Server.” Generally, my friends thought this was too expensive.

After this happened a few times I had a revelation — if I could sell the same program to, say, three people, I could charge $2000 and come out ahead. Or thirty people for $200. Software is neat like that. So in late 2000, Michael sat down, ported the code so that it worked on Access or SQL Server, pulled all the site-specific stuff out into a header file, and we started selling the thing. I didn’t really expect much to come of it.

In those days, I thought, golly, there are zillions of bug tracking packages out there. Every programmer has written a dinky bug tracking package. Why would anyone buy ours? I knew one thing: programmers who start businesses often have the bad habit of thinking everybody else is a programmer just like them and wants the same stuff as them, and so they have an unhealthy tendency to start businesses that sell programming tools. That’s why you see so many scrawny companies hawking source-code-generating geegaws, error catching and emailing geegaws, debugging geegaws, syntax-coloring editing tchotchkes, ftping baubles, and, ahem, bug tracking packages. All kinds of stuff that only a programmer could love. I had no intention of falling into that trap!

Of course, nothing ever works out exactly as planned. FogBUGZ was popular. Really popular. It accounts for a significant chunk of Fog Creek’s revenue and sales are growing steadily. The People won’t stop buying it.

So we did version 2.0. This was an attempt to add some of the most obviously needed features. While David worked on version 2.0 we honestly didn’t think it was worth that much effort, so he tended to do things in what you might call an “expedient” fashion rather than, say, an “elegant” fashion. Certain, ahem, design issues in the original code were allowed to fester. There were two complete sets of nearly identical code for drawing the main bug-editing page. SQL statements were scattered throughout the HTML hither and yon, to and fro, pho and ton. Our HTML was creaky and designed for those ancient browsers that were so buggy they could crash loading about:blank.

Yeah, it worked brilliantly, we’ve been at zero known bugs for a while now. But inside, the code was, to use the technical term, a “big mess.” Adding new features was a hemorrhoid. To add one field to the central bug table would probably require 50 modifications, and you’d still be finding places you forgot to modify long after you bought your first family carplane for those weekend trips to your beach house on Mars.

A lesser company, perhaps one run by an executive from the express-package-delivery business, might have decided to scrap the code and start over.

Did I mention that I don’t believe in starting from scratch? I guess I talk about that a lot.

Anyway, instead of starting from scratch, I decided it was worth three weeks of my life to completely scrub the code. Rub a dub dub. In the spirit of refactoring, I set out a few rules for this exercise.

  1. No New Features, not even small ones, would be added.
  2. At any time, with any check in, the code would still work perfectly.
  3. All I would do is logical transformations — the kinds of things that are almost mechanical and which you can convince yourself immediately will not change the behavior of the code.

I went through each source file, one at a time, top to bottom, looking at code, thinking about how it could be better structured, and making simple changes. Here are some of the kinds of things I did during these three weeks:

  • Changed all HTML to xhtml. For example, <BR> became <br />, all attributes were quoted, all nested tags were matched up, and all pages were validated.
  • Removed all formatting (<FONT> tags etc.) and put everything in a CSS style sheet.
  • Removed all SQL statements from the presentation code and indeed all program logic (what the marketing types like to call “business rules”). This stuff went into classes which were not really designed — I simply added methods lazily as I discovered a need for them. (Somewhere, someone with a big stack of 4×6 cards is sharpening their pencil to poke my eyes out. What do you mean you didn’t design your classes?)
  • Found repeated blocks of code and created classes, functions, or methods to eliminate the repetition. Broke up large functions into multiple smaller ones.
  • Removed all remaining English language text from the main code and isolated that in a single file for easy internationalization.
  • Restructured the ASP site so there is a single entry point instead of lots of different files. This makes it very easy to do things that used to be hard, for example, now we can display input error messages in the very form where the invalid input occurred, something which should be easy if you lay things out right, but I hadn’t laid things out right when I was learning ASP programming way back when.

Over three weeks the code got better and better internally. To the end user, not too much changed. Some fonts are a little nicer thanks to CSS. I could have stopped at any point, because at any given time I had 100% working code (and I uploaded every checkin to our live internal FogBUGZ server to make sure). And in fact I never really had to think very hard, and I never had to design a thing, because all I was doing was simple, logical transformations. Occassionally I would encounter a weird nugget in the code. These nuggets were usually bug fixes that had been implemented over the years. Luckily I could keep the bug fix intact. In many of these cases, I realized that had I started from scratch, I would have made the same bug all over again, and may not have noticed it for months or years.

I’m basically done now. It took, as planned, three weeks. Almost every line of code is different now. Yep, I looked at every line of code, and changed most of them. The structure is completely different. All the bug-tracking functionality is completely separate from the HTML UI functionality.

Here are all the good things about my code scrubbing activity:

  • It took vastly less time than a complete rewrite. Let’s assume (based on how long it took us to get this far with FogBUGZ) that a complete rewrite would have taken a year. Well, that means I saved 49 weeks of work. Those 49 weeks represent knowledge in the design of the code that I preserved intact. I never had to think, “oh, I need a new line here.” I just had to change <BR> to <br /> mindlessly and move on. I didn’t have to figure out how to get multipart working for file uploads. It works. Just tidy it up a bit.
  • I didn’t introduce any new bugs. A couple of tiny ones, of course, probably got through. But I was never doing the types of things that cause bugs.
  • I could have stopped and shipped at any time if necessary.
  • The schedule was entirely predictable. After a week of this, you can calculate exactly how many lines of code you clean in an hour, and get a darn good estimate for the rest of the project. Try that, Mozilla river-drawing people.
  • The code is now much more amenable to new features. We’ll probably earn back the three weeks with the first new major feature we implement.

Much of the literature on refactoring is attributable to Martin Fowler, although of course the principles of code cleanups have been well known to programmers for years. An interesting new area is refactoring tools, which is just a fancy word for programs that do some of this stuff automatically. We’re a long way from having all the good tools we need — in most programming environments you can’t even do a simple transformation like changing the name of a variable (and having all the references to it change automatically). But it’s getting better, and if you want to start one of those scrawny companies hawking programming tool geegaws, or make a useful contribution to open source, the field is wide open.