Posts by Joel Spolsky

In 2000 I co-founded Fog Creek Software, where we created lots of cool things like the FogBugz bug tracker, Trello, and Glitch. I also worked with Jeff Atwood to create Stack Overflow and served as CEO of Stack Overflow from 2010-2019. Today I serve as the chairman of the board for Stack Overflow, Glitch, and HASH.

Picking a Ship Date

One of the best reasons to make a detailed schedule is because it gives you an excuse to cut features. If there’s no way to make the ship date and implement Bob’s Singalong MP3 Chat feature, it’s easy to cut that feature without making Bob feel bad.

So my basic rule for software release cycles is:

  1. Set a ship date, which might as well be arbitrary
  2. Make a list of features and sort them out by priority
  3. Cut low-priority features every time you slip so as to make the date.

Box FactoryIf you do this well, you’ll soon discover that you don’t regret the features that you cut. They’re usually kind of dumb. If they were so important, you can do them next time. It’s like editing. If you ever want to write a brilliant 750 word essay, start by writing a 1500 word essay and then edit.

One way to screw this up, by the way, is forgetting to do features in order of priority. If you aren’t careful, your programmers are going to do features in order of fun, and you won’t be able to ship or cut features because all the programmers were busy writing Karaoke Recalc before they even got the menus to work, and now here it is, six months after the day you wanted to ship, and you have one hell of an easter egg but no functionality.

So the obvious question is: how do you pick a ship date?

It’s conceivable that you have outside constraints. The stock market is switching from fractions to decimals on such-and-such a date, and if you haven’t got the new software ready, your firm will be forced out of business and you personally will be taken out behind the loading docks and shot in the head. Or maybe a new version of the Linux kernel is coming out soon with yet another all-new system to implement packet filtering; all your customers are getting it; and your existing application won’t run on it. OK, for those people, your ship date is easy to figure out. You can stop reading this article now. Go cook a nice dinner for your loved ones.

Bye, now!

But how should the rest of us pick a ship date?

There are three approaches you could take.

  1. Frequent Small Releases. This is the Extreme Programming approach, and is most appropriate for small-team projects with a small number of customers, such as in-house IT development.
  2. Every 12 to 18 months. This is typical of shrinkwrapped software products, desktop applications, etc., where you have larger teams and thousands or millions of customers.
  3. Every 3-5 years. This is typical of huge software systems and platforms that are worlds unto themselves. Operating systems, .Net, Oracle, and for some reason Mozilla fall into this category. They often have thousands of developers (VS.Net had 50 people on the installer team) and enormously complex interactions to other shipping software which can’t be allowed to break.

Here are some of the things you need to think about when deciding how often to release software.

Short releases get customer feedback fast. Sometimes the best way to work with a customer is to show them code, let them try it out, and immediately incorporate their feedback into a build which you give them the very next day. You won’t waste a year developing a complicated system with lots of features that nobody uses, because you’ll be so busy doing things that customers are requesting right now. If you have a small number of customers, prefer frequent small releases. The size of each release should be the minimum chunk that does something useful.

Several years ago I was assigned to develop a web content management system for MTV. The requirements called for a database-backed system with templates, and a complete workflow system that allowed unpaid MTV stringers at colleges across the country to input information about clubs, record stores, radio stations, and concerts. “How are you building the site now?” I asked.

“Oh, we just do it all manually with BBEdit,” they told me. “Sure, there are thousands of pages, but BBEdit has a really good global-find-and-replace function…”

I figured the whole system would take six months to deliver. “But let me suggest something else. Let’s get the templating stuff working first. I can get you that in 3 months and it will save tons of manual work right away. Once that’s working we’ll start in on the workflow component; in the meantime you can continue to do workflow with email.”

They agreed. It sounded like a great idea. Guess what? Once I delivered the templating feature, they realized that they didn’t really need workflow that much. And the templating turned out to be useful for lots of other web sites which didn’t need workflow, either. So we never built workflow, saving three months which I used to enhance the templating feature which turned out to be more useful.

Jewish Quarter

Some types of customers don’t appreciate being “guinea pigs” in this fashion. Generally, people who buy off-the-shelf software don’t want to be part of a Grand Development Experiment; they want something that anticipates their needs. As a customer, the only thing better than getting feature requests done quickly is getting them instantaneously because they’re already in the product, because it was designed thoughtfully and extensively usability- and beta-tested before being inflicted on the world. If you have (or want) a large number of paying customers, prefer less frequent releases.

If you ship an anemic commercial program just to get something out the door so you can “start listening to customers,” what you’ll hear those customers saying is “it doesn’t do very much,” which you might think is OK. Hey, it’s 1.0. But then if you release 2.0 four months later, everybody’s going to think, “That feeble program? What am I, supposed to keep evaluating it every four months just to see if it’s gotten better yet?!” And in fact five years down the line, people will still remember their first impression of 1.0, and it will be almost impossible to get them to reevaluate. Think about what happened to poor Marimba. They launched their company with infinite VC in the days of hyper-Java-hype, having lured the key developers from the Java team at Sun. They had a CEO, Kim Polese, who was brilliant at public relations; when she was marketing Java she had Danny Hillis making speeches about how Java was the next step in human evolution; George Gilder wrote these breathless articles about how Java was going to completely upturn the very nature of human civilization. Compared to Java, we were to believe, monotheism, for example, was just a wee blip. Polese is that good. So when Marimba Castanet launched it probably had more unearned hype than any product in history, but the developers had only been working on it for a total of … four months. We all downloaded it and discovered that — tada! — it was a list box that downloaded software. (What do you expect from four months of development?) Big whoop. The disappointment was so thick you could cut it with butter. And here it is, six years later, ask anybody what Castanet is and they’ll tell you it’s a list box that downloads software. Hardly anyone bothered to reevaluate it, and Marimba has had six years to write code; I’m sure it’s just the coolest thing now but, honestly, who has time to find out? Let me tell you a little secret: our strategy for CityDesk is to avoid massive PR until 2.0 is out. That’s the version that we want everybody on earth to get their first impressions from. In the meantime we’ll do quiet guerilla marketing, and anybody who finds it will discover that it’s a completely spiffy program that solves a lot of problems, but Arnold Toynbee won’t have to rewrite anything.

[Image]With most commercial software, you’ll discover that the process of designing, prototyping, integrating, fixing bugs, running a full alpha and beta cycles, creating documentation, and so forth takes 6-9 months. In fact if you try to do a full release every year, you only have time for about 3 months worth of new code development. Software that is upgraded annually usually doesn’t feel like it has enough new features to justify the upgrade. (Corel PhotoPaint and Intuit Quickbooks are particularly egregious examples of this; they have a new “major” version every year which is rarely worth buying). As a result many people have learned by now to skip every other release. You don’t want your customers getting in that habit. If you stretch out your schedule to 15 or 18 months between releases, you get 6 months of new features instead of 3 months worth, which makes your upgrade a lot more compelling.

OK, if 15 months is so good, wouldn’t 24 months be better? Maybe. Some companies can get away with it, if they are major leaders in their category. PhotoShop seems to get away with it. But as soon as an application starts to feel old, people stop buying it because they are expecting that new version Any Day Now. This can cause serious cash flow problems for a software business. And of course you may have competitors nipping at your heels.

For large platform software — operating systems, compilers, web browsers, DBMSs — the hardest part of the development process is maintaining compatibility with thousands or millions of existing applications or hardware. When a new version of Windows comes out, you very rarely hear about backwards compatibility problems. The only way they can achieve this is with insane amounts of testing that make the construction of the Panama Canal seem like a weekend do-it-yourself project. Given the typical 3 year cycle between major Windows releases, almost all of that time is spent in the boring integration and testing phase, not writing new features. Releasing new versions any more often than that is just not realistic. And it would drive people crazy. Third party software and hardware developers would simply revolt if they had to test against lots of little incremental releases of an operating system. For Systems With Millions of Customers and Millions of Integration Points, Prefer Rare Releases. You can do it like Apache: one release at the beginning of the Internet Bubble, and one release at the end. Perfect.

Souk - Damascus GateIf you have a lot of validation and unit tests, and if you write your software carefully, you may get to the point where any daily build is almost high enough quality to ship. This is certainly something to strive for. Even if you’re planning the next release for three years from now, the competitive landscape may suddenly change and there may be a good reason to do a quick interim release to react to a competitor. When your wife is about to give birth, it’s not really a good idea to take apart your car’s engine. Instead build a new one on the side and don’t hook it up until it’s perfect.

But don’t overestimate what you can accomplish by keeping high quality daily builds. Even if you are permanently at zero bugs, if your software has to run In The Wild, you’re never going to find all the compatibility bugs and Windows 95 bugs and It-Doesn’t-Work-With-Large-Fonts-Turned-On bugs until you do a few betas, and there is no realistic way to do a beta cycle in less than 8 weeks.

One final thought. If your software is delivered as a service over the web, like ebay or PayPal, theoretically there’s nothing to stop you from frequent small releases, but this might not be the best thing to do. Remember the cardinal rule of usability: an application is usable if it behaves the way that the user expected it to behave. If you keep changing things around every week, it won’t be so predictable, so it won’t be so usable. (And don’t think you can get around this by having one of those annoying screens with a paragraph of text saying Warning! The UI has changed! Nobody reads those.) From a usability perspective a better approach would probably be less frequent releases that include a bunch of changes all at once, where you make an effort to change the visuals of the whole site so that it all looks weird and users intuit that things have changed a lot and they have to be careful.

2002/04/07

Whenever I start to read a review for a new “bloat-free” MS Office competitor, I know what the script is going to be:

  1. Office is overpriced and “bloated” with too many features that you “don’t need”
  2. Luckily, <company X> made a smaller leaner one without all the bloat
  3. But I’m never going to use it, because it doesn’t have <insert my favorite feature> or word count.

It’s actually quite amusing how often this exact script, including the “word count” gripe, recurs.  Here’s one from 2001. And another from 1997.  Oh, look, it happened again.

Remember, kids, the trouble with the “everyone only uses 20% of the features” myth is that everybody uses a slightly different 20%, and the journalist who has to review your exciting new BloatFreeWrite has been told to write a 250 word story.

(For review: Bloatware and the 80/20 myth. No matter how much it bothers you neat freaks, the market always votes for bloatware.)

2002/04/02

Can’t Understand It? Don’t Worry.

Whenever somebody gives you a spec for some new technology, if you can’t understand the spec, don’t worry too much. Nobody else is going to understand it, either, and it’s probably not going to be important. This is the lesson of SGML, which hardly anyone used, until Tim Berners-Lee dumbed it down dramatically and suddenly people understood it. For the same reason he simplified the file transfer protocol, creating HTTP to replace FTP.

You can see this phenomenon all over the place; even within a given technology some things are easy enough to figure out and people use them (like COM’s IUnknown), while others are so morbidly complicated (IMonikers) when they should be simple (what’s wrong with URLs?) that they languish.

2002/03/20

Today I’m learning ADO.NET. Am I crazy, or is this a step backwards?

Adam Fisk: Lose the Joel-Worship!

It’s true. I get email from people on development teams who appear to be in some kind of big fight over something, and they are hitting each other over the head with various quotes from me, instead of thinking for themselves, and now they want me to adjudicate, as if I know the first thing about their problems or their world. I haven’t yet written the article that says that if you can’t think for yourself, no amount of “methodology” is going to save you.

2002/03/18

Sydney Morning Herald: “Countless managers of small to medium sites have wondered how to cross the chasm from hand-built pages to a true content management system. Now a product called CityDesk provides the best answer yet.” (Reprinted here)

From the Flogging of Dead Horses department
Rick Chapman interviewed me again… no, I still don’t think you should rewrite large software products from the ground up. Not even Outlook.

2002/03/14

Three months after the original version, Fog Creek Software shipped a new version of CityDesk today, a free service pack release that fixes about 100 bugs and adds some spiffy new features that people were clamoring for. The biggest complaint about CityDesk was that it generated weird URL names (which actually surprised me, because most other content management systems do the same). So we fixed that. We added a few easy features that I think will go a long way. And we fixed all known bugs — many of which we found out about because CityDesk catches most crashes and transmits crucial crash info to us.

What’s New in SP1

2002/03/13

As promised (months ago!) I’ve completely revamped my list of recommended books. I’ve always wanted this to be a “programmers bookshelf” listing the books that I think every software developer should read. So I removed some books, good ones but not crucial ones, and I added a bunch of books, and came up with 19 must-read books (plus 2 books if you want to become a programmer). Visit the new Book Reviews page.

Book Reviews

 

Nothing is as Simple as it Seems

We had a little usability problem in CityDesk.

Here was the problem: you could import files from the web using a menu command (“Import Web Page”). And you could import files from a disk into CityDesk by dragging them with the mouse. But there was no menu command to import files from a disk. So either people didn’t discover that it was possible, or people tried to use the Import Web Page feature to import from disk, which didn’t work right.

I thought that it would be easy to fix with a two page wizard. Roughly speaking, page one of the wizard would ask you “Where do you want to import from?” If you chose “disk,” page two would prompt you for a file. If you chose “web,” page two would prompt you for a URL.

I almost started implementing this, but something stopped me, and instead, I started to write a mini-spec. Here’s the spec, in its entirety:

Page One
Where do you want to import from? (Disk/Web)

Page Two (Disk)
Standard File/Open dialog

Page Two (Web)
URL prompt with mini-web-browser

Suddenly something occurred to me. Can you put the Windows standard file open dialog, which is usually supplied in toto by the OS, into a wizard?

Hmm.

I investigated. Yes, you can, but it’s no fun and takes a few hours of work. How could I make this not be a wizard? I rewrote the spec:

Two Menu Items:
1) Import Web Page From Internet -> Pops Up URL Dialog
2) Import Web Page From Disk -> Pops Up File Open Dialog

Much better. Three minutes of design work saved me hours of coding.

If you’ve spent more than 20 minutes of your life writing code, you’ve probably discovered a good rule of thumb by now: nothing is as simple as it seems.

Something as simple as copying a file is full of perils. What if the first argument is a directory? What if the second argument is a file? What if a file with the same name already exists in the destination? What if you don’t have write permission?

What if the copy fails in the middle? What if the destination is on a remote machine which is available, but which requires authentication to continue? What if the files are large and the link is slow and you need to show a progress indicator? What if the transfer speed slows down to almost zero… when do you give up and return an error message?

A good way to interview candidates for testing jobs is to give them a simple operation and ask them to enumerate all the things that can possibly go wrong. A classic Microsoft test interview question: how do you test the File Open dialog box? A good tester will be able to rattle off several dozen weird things to test (“file is deleted by another user between the time it is listed in the box and the time you select it and click Open“).

OK, so we have one axiom: nothing is as simple as it seems.

There’s another axiom in software engineering, which is, always try to reduce risk. One particularly important piece of risk to avoid is schedule risk, when something takes longer than expected. Schedule risk is bad because your boss yells at you, which makes you unhappy. If that’s not enough motivation for you, the economic reason why schedule risk is bad is because you decided to do a feature based on information that it would take 1 week. Now that you realize that it has taken 20 weeks to accomplish, that decision might well have been wrong. Perhaps if you knew it was going to take 20 weeks, you would have made a different decision. The more wrong decisions you make, the more likely all those tote bags with your company logo will end up in the liquidator’s warehouse while your ex-CEO mopes that “what sucks is, we weren’t even successful enough to get mentioned on f***edcompany.com when we shut down!”

The combination of nothing-is-as-simple-as-it-seems and reduce-risk can only lead you to one conclusion:

You have to design things before you implement them.

I’m sorry to disappoint you. Yeah, I know, you read Kent Beck and now you think it’s OK to not design things before you implement them. Sorry, it’s not OK. You can not change things in code “just as easily” as you could change them in the design documents. People say this all the time and it’s wrong. “We use high-level tools these days, like Java and XML. We can change things in minutes in the code. Why not design it in code?” My friend, you can put wheels on your mama but that doesn’t make her a bus, and if you think you can refactor your wrongly-implemented file-copy function to make it preemptive rather than threaded as quickly as I could write that sentence, you’re in deep denial.

Anyway, I don’t think Extreme Programming really advocates zero design. They just say “don’t do any more design than needed,” which is fine. But that’s not what people hear. Most programmers are looking for any excuse they can find not to do basic design before implementing features. So they latch onto the “no-design” idea like flies in a bug zapper. Dzzzzzzt! It’s one of those weird forms of laziness where you end up doing more work than you would have done otherwise. I’m too lazy to design the feature on paper first, so I just write some code, and then it’s not right, so I have to fix it, and I spend more time than I would have otherwise. Or, more commonly, I write some code, and then it’s not right, but it’s too late, and my product is inferior and I spend until the end of time making up excuses for why it “has to be that way.” It’s just sloppy and unprofessional.

When Linus Torvalds bashes design, he’s talking about huge systems, which have to evolve, or they become Multics. He’s not talking about your File Copy code. And when you consider that he had a pretty clear road map of exactly where he was going, it’s no wonder Linus doesn’t see much value in design. Don’t fall for it. Chances are it doesn’t apply to you. And anyway, Linus is much smarter than we are, so things that work for him don’t work for us normal people.

Incremental design and implementation is good. Frequent releases are fine (although for shrink-wrapped or mass market software, it drives customers crazy, never a good idea — instead do frequent internal milestones.) Too much formality in design is a waste of time — I’ve never seen a project benefit from mindless flowcharting or UMLing or CRCing or whatever the flavor-du-jour is. And those huge 10 million lines-of-code behemoth systems Linus is talking about should evolve, because humans don’t really know how to design software on that scale.

But when you sit down to write File Copy, or when you sit down to plan the features of the next release of your software, you gotta design. Don’t let the sirens persuade you otherwise.

Come argue with me about this in person at the Cutter Summit, April 29-May 1st, in Cambridge, MA, where I’ll be on a panel with Tom DeMarco, Kent Beck, and others.