Category / Reading Lists

by Joel Spolsky

What is the Work of Dogs in this Country?

How naive were we?

We had assumed that Bezos was just reinvesting the profits, that’s why they weren’t showing up on the bottom line.

Last year, about this time, the first big dotcom failures started to hit the news. Boo.com. Toysmart.com. The Get Big Fast mentality was not working. Five hundred 31-year-olds in Dockers discovered that just copying Jeff Bezos wasn’t a business plan.

The past few weeks have felt oddly quiet at Fog Creek. We’re finishing up CityDesk. I’d like to tell you all about CityDesk, but that will have to wait. I need to tell you about dog food.

Dog food?

Last month Sara Corbett told us about the Lost Boys, Sudanese refugees between 8 and 18 years old separated from their families and forced on a thousand mile march from Sudan, to Ethiopia, to Sudan, to Kenya. Half died on that trip, of hunger, thirst, alligators. A few of them were rescued and delivered to places like Fargo, North Dakota, in the middle of winter. “Are there lions in this bush?” one asked, riding in a car to his new home from the airport.

Peter touched my shoulder. He was holding a can of Purina dog food. ”Excuse me, Sara, but can you tell me what this is?” Behind him, the pet food was stacked practically floor to ceiling. ”Um, that’s food for our dogs,” I answered, cringing at what that must sound like to a man who had spent the last eight years eating porridge. ”Ah, I see,” Peter said, replacing the can on the shelf and appearing satisfied. He pushed his grocery cart a few more steps and then turned again to face me, looking quizzical. ”Tell me,” he said, ”what is the work of dogs in this country?” [New York Times Magazine, April 1, 2001]

Dogs. Yes, Peter. Fargo has enough food, even for dogs.

It’s been a depressing year.

Oh, it started out so amusing, we all piled into B2B and B2C and P2P like a happy family getting in the Suburban for a Sunday outing to the Krispy Kreme Donut Shop. But wait, that’s not even the amusing part, the amusing part was watching the worst business plans fail, as their stock went from 316 to 3/16. Take that, new economy blabbermouths! Ah, the schadenfreude. Ah, the glee, when once again, Wired Magazine proves that as soon as it puts something on the cover, that thing will be proven to be stupid and wrong within a few short months.

Ooh, sorry, did you buy the Wired Index?

And with this New Economy thing, Wired really blew it, because they should have known by then what a death kiss their cover was for any technology or company or meme, after years of touting smell-o-rama and doomed game companies and how PointCast was going to replace the web, no, wait, PointCast already replaced the web, in March 1997.  But they tempted fate anyway, and didn’t just put the New Economy on the cover, they devoted the whole goddamn issue to the New Economy, thus condemning the NASDAQ to plummet like a sheep learning to fly.

picture-small-goats:

But joy at other’s misfortune can only entertain us for so long. Now it’s just getting depressing, and I know the economy is not officially in a depression, but I’m depressed, not because so many stupid startups went away, but because the zeitgeist is depressing. And now we have to eat dog food instead of Krispy Kremes.

Which is what we’re doing, because life goes on. Even though everybody’s walking around with their chins glued to their chests, mourning about the hours they devoted, ruining their health and love lives for the sake of stock options in SockPuppet.com, life goes on. And the product development cycle must go on, and we at Fog Creek are getting towards the part in the product development cycle where you have to eat your own dog food. So for a while we’re Dog Creek Software.

Eating your own dog food is the quaint name that we in the computer industry give to the process of actually using your own product. I had forgotten how well it worked, until a month ago, I took home a build of CityDesk (thinking it was about 3 weeks from shipping) and tried to build a site with it.

Phew! There were a few bugs that literally made it impossible for me to proceed, so I had to fix those before I could even continue. All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn’t uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute.

And not just those. As I worked, not even exercising the features, just quietly trying to build a simple site, I found 45 bugs on one Sunday afternoon. And I am a lazy man, I couldn’t have spent more than 2 hours on this. I didn’t even try anything but the most basic functionality of the product.

Monday morning, when I got in to work, I gathered the team in the kitchen. I told them about the 45 bugs. (To be fair, many of these bugs weren’t actual defects but simply things that were not as convenient as they should have been). Then I suggested that everybody build at least one serious site using CityDesk to smoke out more bugs. That’s what is meant by eating your own dog food.

Here’s one example of the kind of things you find.

I expect that a lot of people will try to import existing web pages into CityDesk by copying and pasting HTML code. That works fine. But when I tried to import a real live page from the New York Times, I spent a whole day patiently editing the HTML, finding all the IMG links (referring to outside pictures), downloading the pictures from the web, importing those pictures into CityDesk, and adjusting the IMG links to refer to the internal pictures. It’s hard to believe, but one article on that web site contains about 65 IMG links referring to 35 different pictures, some of which are 1 pixel spacers which are very difficult to download using a web browser. And CityDesk has a funny compulsion to change the name of imported pictures internally into a canonical number, and it doesn’t even have a way to find out what that number is, so the long and the short of it was that it took me one full day to import a page into CityDesk.

It was getting a bit frustrating, so I went and weeded the garden for a while. (I don’t know what we’ll do to relieve stress when it’s all cleaned up. Thank God we can’t afford a landscaping service.) And that’s when it hit me. Hey, I’m a programmer! In the time it took me to import one page and adjust the pictures, I could write a subroutine that does it automatically! In fact, it probably took me less time to write the subroutine. Now importing a page takes about half a minute, instead of one day, and is basically error-free.

Wow.

That’s why you eat your own dog food.

And when Michael started importing some sites himself, he found about 10 bugs I had baked in by mistake. For example, we found web sites that use complicated names for pictures that cannot be converted to file names when you import them because they contain question marks, which are legal in URLs but not legal in file names.

Sometimes you download software and you just can’t believe how bad it is, or how hard it is to accomplish the very simple tasks that the software tries to accomplish. Chances are, it’s because the developers of the software don’t use it.

I have an even more amusing example of failing to eat dog food. Guess what email product was used internally at Juno Online Services? [If you’re just tuning in, I worked on the Juno client team for a few years].

Hmm, did you guess Juno? Since that was, uh, our product?

No. A couple of people, including the president, used Juno at home. The other 175 of us used Microsoft Outlook.

And for good reasons! The Juno client was just not such a great email client; for two years the only thing we worked on was better ways to show ads. A lot of us thought that if we had to use the product, we would have to make it better, if only to stop our own pain. The president was very insistent that we show popup ads at six different points in time, until he got home and got six popup ads, and said, “You know what? Maybe just two popups.”

AOL was signing up members at a furious rate, in part because it provided a better user experience than Juno, and we didn’t understand that, because we didn’t eat our own dog food. And we didn’t eat our own dog food, because it was disgusting, and management was sufficiently dysfunctional that we simply were not allowed to fix it, and at least make it tolerable to eat.

Anyway. CityDesk is starting to look a lot better. We’ve fixed all those bugs, found some more, and fixed them, too. We’re adding features we forgot about that became obviously necessary. And we’re getting closer to shipping! Hurrah! and thankfully, we no longer have to contend with 37 companies, each with $25 million in VC, competing against us by giving away their product for free in exchange for agreeing to have a big advertisement tattooed on your forehead. In the post-new economy, everybody is trying to figure out how much they can get away with charging. There’s nothing wrong with the post-new economy, if you’re smart. But all the endless news about the “dot coma” says more about the lack of creativity of business press editors than anything else. Sorry, fuckedcompany.com, it was funny for a month or so, now it’s just pathetic. We’ll focus on improving our product, and we’ll focus on staying in business, by listening to our customers and eating our own dog food, instead of flying all over the country trying to raise more venture capital.

by Joel Spolsky

Don’t Let Architecture Astronauts Scare You

When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. Blah.

When you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don’t know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don’t actually mean anything at all. 

These are the people I call Architecture Astronauts. It’s very hard to get them to write code or design programs, because they won’t stop thinking about Architecture. They’re astronauts because they are above the oxygen level, I don’t know how they’re breathing. They tend to work for really big companies that can afford to have lots of unproductive people with really advanced degrees that don’t contribute to the bottom line.

A recent example illustrates this. Your typical architecture astronaut will take a fact like “Napster is a peer-to-peer service for downloading music” and ignore everything but the architecture, thinking it’s interesting because it’s peer to peer, completely missing the point that it’s interesting because you can type the name of a song and listen to it right away.

All they’ll talk about is peer-to-peer this, that, and the other thing. Suddenly you have peer-to-peer conferences, peer-to-peer venture capital funds, and even peer-to-peer backlash with the imbecile business journalists dripping with glee as they copy each other’s stories: “Peer To Peer: Dead!”

The Architecture Astronauts will say things like: “Can you imagine a program like Napster where you can download anything, not just songs?” Then they’ll build applications like Groove that they think are more general than Napster, but which seem to have neglected that wee little feature that lets you type the name of a song and then listen to it — the feature we wanted in the first place. Talk about missing the point. If Napster wasn’t peer-to-peer but it did let you type the name of a song and then listen to it, it would have been just as popular.

Another common thing Architecture Astronauts like to do is invent some new architecture and claim it solves something. Java, XML, Soap, XmlRpc, Hailstorm, .NET, Jini, oh lord I can’t keep up. And that’s just in the last 12 months!

I’m not saying there’s anything wrong with these architectures… by no means. They are quite good architectures. What bugs me is the stupendous amount of millennial hype that surrounds them. Remember the Microsoft Dot Net white paper? 

The next generation of the Windows desktop platform, Windows.NET supports productivity, creativity, management, entertainment and much more, and is designed to put users in control of their digital lives.

That was about 9 months ago. Last month, we got Microsoft Hailstorm. That white paper says:

People are not in control of the technology that surrounds them….HailStorm makes the technology in your life work together on your behalf and under your control.

Oh, good, so now the high tech halogen light in my apartment will stop blinking randomly.

Microsoft is not alone. Here’s a quote from a Sun Jini whitepaper:

These three facts (you are the new sys admin, computers are nowhere, the one computer is everywhere) should combine to improve the world of using computers as computers — by making the boundaries of computers disappear, by making the computer be everywhere, and by making the details of working with the computer as simple as putting a DVD into your home theater system.

And don’t even remind me of the fertilizer George Gilder spread about Java:

A fundamental break in the history of technology…

That’s one sure tip-off to the fact that you’re being assaulted by an Architecture Astronaut: the incredible amount of bombast; the heroic, utopian grandiloquence; the boastfulness; the complete lack of reality. And people buy it! The business press goes wild!

Why the hell are people so impressed by boring architectures that often amount to nothing more than a new format on the wire for RPC, or a new virtual machine? These things might be good architectures, they will certainly benefit the developers that use them, but they are not, I repeat, not, a good substitute for the messiah riding his white ass into Jerusalem, or world peace. No, Microsoft, computers are not suddenly going to start reading our minds and doing what we want automatically just because everyone in the world has to have a Passport account. No, Sun, we’re not going to be able to analyze our corporate sales data “as simply as putting a DVD into your home theatre system.”

Remember that the architecture people are solving problems that they think they can solve, not problems which are useful to solve. Soap + WSDL may be the Hot New Thing, but it doesn’t really let you do anything you couldn’t do before using other technologies — if you had a reason to. All that Distributed Services Nirvana the architecture astronauts are blathering about was promised to us in the past, if we used DCOM, or JavaBeans, or OSF DCE, or CORBA.

It’s nice that we can use XML now for the format on the wire. Whoopee. But that’s about as interesting to me as learning that my supermarket uses trucks to get things from the warehouse. Yawn. Mangos, that’s interesting. Tell me something new that I can do that I couldn’t do before, O Astronauts, or stay up there in space and don’t waste any more of my time.

by Joel Spolsky

How Many Lies Can You Find In One Direct Mail Piece?

George Meyer, one of the writers on the Simpsons, likes to collect examples of advertising in which “the word-to-falsehood ratio approaches one.” In an interview in the New Yorker, he complained about a magazine ad for a butter substitute called Country Crock. “It’s not from the country, there is no crock,” he told the interviewer. “Two words, two lies.”

Recently I got this piece of junk mail from Earthlink (larger view)

Just for my own personal amusement, I decided to try to count how many lies are on this particular fine specimen.

  1. The whole piece is attempting to disguise itself as an express mail letter. They are working on the assumption that ignorant recipients will confuse it with a Federal Express letter, or perhaps think that “Urgent Express” is a company that competes with FedEx. Uh huh. Ooooh! The marketing slimeballs at Earthlink must have drooled at the thought of old Marge in a nursing home, practically falling over with excitement that she got an urgent express letter, like those letters you get from fancy New York lawyers urging you to sue the nursing home.

  2. It says that it’s urgent, and it isn’t, it’s just a stupid CD trying to get you to sign up for Internet access.

  3. It says that it’s “express”, but it’s mailed using “Presorted Standard” rate, which isn’t express at all, in fact, it probably cost precisely $0.23 to mail (FedEx is $15.34).

  4. It has a tracking number, but it is using a class of mail service that is not tracked, and I’ll bet you anything that everybody gets the same tracking number. Sorry, folks, for $0.23 you don’t get tracking.

  5. It uses a fake handwriting font, intended to make you think that it is not junk mail but rather something that a person made for you.

  6. It says that it’s from Sky Dayton (founder of Earthlink) although it is actually from Earthlink’s direct marketing department. I can just see the thought going through the copy writer’s head. “People won’t open it unless it’s from someone they know, and everyone knows Sky Dayton!”

  7. It has a “sender’s account number” which is, of course, completely fake (Anybody care to try to use it to sign up for Earthlink?)

  8. It has a little “payment” section with a checkbox indicating that it should be billed to the sender, even though this has no meaning on standard mail.

  9. It has a “release signature” with a little annotation that this signature is “required,” when actually no “release signature” is required at all.

  10. It says “Time Sensitive Material Enclosed,” which is almost certainly untrue and meaningless anyway.

  11. On the back, it says “Weight Limit 8 Ounces” even though the actual weight limit for Presorted Standard is 16 Ounces.

  12. It actually has a piece of clear plastic glued on the front. The clear plastic serves no purpose whatsoever except to imitate FedEx envelopes which have the same plastic to hold the mailing manifest, much like those little butterflies which have fierce markings in attempt to convince butterfly-eaters that they themselves are fierce, even though they couldn’t be less fierce if they were dressed up like Julie Andrews in high heels singing “My Favorite Things.” In nature this is called Batesian mimicry.

The biggest problem with direct mail (what we normal people call “Junk Mail”) is that 99% of it (literally) gets thrown away unopened, Batesian mimicry or not. To combat this, direct mailers will do anything to get you to open their junk, no matter how dishonest.

When I went to Earthlink’s home page, it only took me a couple of clicks to find their Mission and Core Values and Beliefs. And what it says there is:

We require complete honesty and integrity in everything we do.

I didn’t even have the energy to open the damn thing and see how many lies were on the inside. Does this mailing strike you as “completely honest”? Or just “business as usual?” Am I too sensitive? Should I go back to complaining about complaining about Bloatware?

[Followup: Earthlink is unrepentant.]

by Joel Spolsky

Strategy Letter IV: Bloatware and the 80/20 Myth

Version 5.0 of Microsoft’s flagship spreadsheet program Excel came out in 1993. It was positively huge: it required a whole 15 megabytes of hard drive space. In those days we could still remember our first 20MB PC hard drives (around 1985) and so 15MB sure seemed like a lot.

By the time Excel 2000 came out, it required a whopping 146MB … almost a tenfold increase! Dang those sloppy Microsoft programmers, right?

Wrong.

I’ll bet you think I’m going to write one of those boring articles you see all over the net  bemoaning “bloatware”.  Whine whine whine, this stuff is so bloated, oh woe is me, edlin and vi are so much better than Word and Emacs because they are svelte, etc.

Ha ha! I tricked you! I’m not going to write that article again, because it’s not true.

In 1993, given the cost of hard drives in those days, Microsoft Excel 5.0 took up about $36 worth of hard drive space.

In 2000, given the cost of hard drives in 2000, Microsoft Excel 2000 takes up about $1.03 in hard drive space.

(These figures are adjusted for inflation and based on hard drive price data from here.)

In real terms, it’s almost like Excel is actually getting smaller!

What is bloatware, exactly? The Jargon File snidely defines it as “software that provides minimal functionality while requiring a disproportionate amount of diskspace and memory. Especially used for application and OS upgrades. This term is very common in the Windows/NT world. So is its cause.”

I guess those guys just hate Windows. I haven’t run out of memory in more than a decade, ever since virtual memory appeared in Windows 386 (1989). And hard drive space is down to $0.0071 per megabyte and still plummeting like a sheep learning to fly by jumping out of a tree.

Maybe Linus Åkerlund can explain it. On his web page, he writes, “The big disadvantage of using these bloated programs is that you have to load this very large program, even if you just want to accomplish one tiny little thing. It eats up all your memory… you’re not using your system in an efficient way. You make the system seem more inefficient than it really is, and this is totally unnecessary.”

Ohhh. It eats up all your memory. I see. Actually, well, no, it doesn’t. Ever since Windows 1.0, in 1987, the operating system only loads pages as they are used. If you have a 15MB executable and you only use code that spans 2MB worth of pages, you will only ever load 2MB from disk to RAM. In fact if you have a modern version of Windows, the OS will automatically rearrange those pages on the hard drive so that they’re consecutive, which makes the program start even faster next time.

And I don’t think anyone will deny that on today’s overpowered, under-priced computers, loading a huge program is still faster than loading a small program was even 5 years ago. So what’s the problem?

RA Downes gives us a clue. It looks like he spent hours dissecting a small Microsoft utility, apparently enraged that it was a whole megabyte in size. (That’s 3.15 cents of hard drive space at the time he wrote the article). In his opinion, the program should have been around 95% smaller. The joke is that the utility he dissected is something called RegClean, which you’ve probably never heard of. This is a program that goes through your Windows registry looking for things that aren’t being used and deleting them. You have to be a little bit on the obsessive-compulsive side to care about cleaning up unused parts of your registry. So I’m starting to suspect that fretting about bloatware is more of a mental health problem than a software problem.

In fact there are lots of great reasons for bloatware. For one, if programmers don’t have to worry about how large their code is, they can ship it sooner. And that means you get more features, and features make your life better (when you use them) and don’t usually hurt (when you don’t). If your software vendor stops, before shipping, and spends two months squeezing the code down to make it 50% smaller, the net benefit to you is going to be imperceptible. Maybe, just maybe, if you tend to keep your hard drive full, that’s one more Duran Duran MP3 you can download. But the loss to you of waiting an extra two months for the new version is perceptible, and the loss to the software company that has to give up two months of sales is even worse.

A lot of software developers are seduced by the old “80/20” rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. 

Unfortunately, it’s never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release “lite” word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the “word count” feature which they need because most journalists have precise word count requirements, and it’s not there, because it’s in the “80% that nobody uses,” and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can’t use this damn thing ’cause it won’t count my words. If I had a dollar for every time this has happened I would be very happy.

When you start marketing your “lite” product, and you tell people, “hey, it’s lite, only 1MB,” they tend to be very happy, then they ask you if it has their crucial feature, and it doesn’t, so they don’t buy your product.

Bottom line: if your strategy is “80/20”, you’re going to have trouble selling software. That’s just reality. This strategy is as old as the software industry itself and it just doesn’t pay; what’s surprising is how many executives at fast companies think that it’s going to work.

Jamie Zawinski says it best, discussing the original version of Netscape that changed the world. “Convenient though it would be if it were true, Mozilla [Netscape 1.0] is not big because it’s full of useless crap. Mozilla is big because your needs are big. Your needs are big because the Internet is big. There are lots of small, lean web browsers out there that, incidentally, do almost nothing useful. But being a shining jewel of perfection was not a goal when we wrote Mozilla.”

by Joel Spolsky

Spring in Cambridge

Even though the weather is in the 40s, and the wind chill is wintry, today I woke up and went outside and felt like spring was finally in the air. Maybe I started to notice this last Friday when I went back to the Fog Creek garden in a t-shirt and wasn’t actually cold. 

Spring conjures up a funny set of associations for me. I think about the fact that it’s still daylight when you go outside after dinner. I think about living in a nice old messy house with real wood floors, lots and lots of books, and flowers in the garden. I think about two of my favorite places in the spring, Berkeley and Boston, cities that revolve around learning and education and the unbridled enthusiasm of youth and the belief that anything is possible. As a first year student I arrived at university without even the basic premise of the shape my life would take, but compared to the unspeakably horrible time I had doing compulsory military duty, any shape of life sounded utopian to me. I could be in theatre! I could write for the school newspaper! Politics! Hacking! Teaching! Art! I could become a competitive swimmer! Play piano! All possible!

Around 1993, I think, I finally got access to the World Wide Web at home. $35 a month for SLIP from Panix. One of the first things that captivated me was Travels with Samantha, by Philip Greenspun. The pictures were almost impossible to make out in 16 VGA colors; the connection was at 14.4 baud; the screen was 640×480. But Greenspun’s style of personal journal was captivating. He told us how lonely it was to live in Las Alamos for three months. He told us how to build an online community. He told us about his $5000 Viking stove.

Greenspun has been a key inspiration to many of us. He started a company called arsDigita, which was really just another consulting company for the Internet, but it had two things which made it uniquely different.

It had a personal voice. When you went to the home page, there was a happy little note about the new offices or the latest course offerings; the style was not boring corporate/marcomm happy-talk, it was Philip telling you that if you’re poor, there are some free services, but if you’re rich, head on over here and we’ll build you a nice service of your own, but bring a lot of money please, because the future’s so bright you gotta wear shades.

And it had education. The web was new and exciting and offered unlimited possibilities. Everybody wanted to learn about it and arsDigita would teach you, for free, starting with a book that cost about $15 more than it should because it was stuffed with completely irrelevant full-color photographs that seemed to be there solely to assuage the author’s egotistical photography hobby, but which really were so bright and colorful and optimistic that nobody could read that book (or any of the photography-filled pages of Greenspun’s websites) without becoming optimistic and excited about the future of the Internet, and when you did, there was a whole chapter on how to start a consulting company just like Philip and get rich off of the Internet, complete with suggested prices and detailed calculations of how you will make $250,000 a year and be able to buy $5000 Viking stoves and a beautiful condo in Harvard Square, in the spring, with flowers budding and optimist young students everywhere practically popping out of their tanktops. Stream of consciousness, random, and interspersed with completely random bogus snippets of half-baked Oracle SQL statements but goddamn it, there was a personal voice there.

(I know, anybody at arsDigita will tell you “no, Joel! It’s open source that made arsDigita different.” Or community. Or content management software. Sorry, all that stuff bores me. What excites me was the way arsDigita had a personal voice on the Internet that made it possible to relate to it in a human way, which is what people want to do, since we’re humans.)

Fog Creek Software was inspired in no small part by arsDigita; the code name for the company was “PaxDigita” and we took inspiration from Greenspun’s tongue-in-cheek corporate slogan “We don’t have venture capital; we have money.” ArsDigita thought that it was enough to be profitable. 

There were two things I thought that arsDigita did wrong. First of all, they didn’t understand that consulting doesn’t scale so well; if you really want to make an impact, you need to license software. A consultant working on a project makes ten or twenty percent profit. If you license software, each marginal unit you sell is 100% profit. That’s why software companies have valuations that are something like 25 times richer than consulting companies. Consulting is important and keeps us in touch with customers, and it funds the software development, but our goal is to be a software company. (ArsDigita seems to have learned this lesson. Allen Shaheen, the CEO, writes that “we are also considering developing and marketing additional software products using a different type of licensing arrangement. This investment is in addition to our investment in ACS as an open source product. We are considering this approach because it would allow us to accelerate our development of new and expanded features.” That’s happy talk for “the new features are going to cost you from now on.”)

The other thing I didn’t like about arsDigita was an almost religious aversion to Microsoft. There was a deep belief that arsDigita “eliminated risk” by never using a Microsoft product. The truth is, they just didn’t know anything about Microsoft products. Their religious bigotry struck me as just that: bigotry. In the words of Lazarus Long, “I don’t believe anything. Gets in the way of learning.” If you say, “we know Unix better, so we get better results with Unix,” you’re a scientist. If you say, “we don’t use any Microsoft crap, it all sucks” you’re just a fanatic, and it’s getting in the way of doing good work. (Greenspun has learned this lesson too; almost all of his bitterness to Microsoft seems to have dissipated and he finally admits, “I’m not motivated to fight a religious war for any particular database management system or Web server.” It’s nice to watch smart people learning.)

But there’s one thing that arsDigita did very, very, right, and that was the personal voice thing. The biggest thing I think about with Fog Creek is how to maintain that personal voice, and if we can do it, I’ll owe a big, big debt to Philip Greenspun.

For ArsDigita, the spring is over. ArsDigita found it impossible to hire engineers during the dotcom bubble without the promises of IPO riches, and took $38 million in venture capital. They grew from 5 to 250 people.

And then the excited exuberance gave way to a completely collapsed market for Internet consulting services. 

The VCs promptly installed boring old-school management who insisted on a happy-talk marcomm website with complete bullshit like “Fully integrated through a single data model to ensure consistency across all business processes, ACS empowers organizations to create Web services that maximize value for the end-user.”

Philip left.

The voice is gone.

To me, arsDigita will always remind me of those silly irrelevant pictures of pretty girls rollerblading and clam stands and boats in the Charles and sunsets in New Mexico and flowers and spring, and if you built it, they will come, and if you have a voice on your website, they will care. When I go to the website today and read that they “maximize value for the end-user,” I think of that dazzling, exciting kid you knew freshman year, who jumped from varsity squash to journalism to drama to campus politics, and was all fired up to change the world. But he got married, has 2 1/2 kids, took a job as an insurance agent and wears a gray suit to work every day.

Maturity is kind of sad.

But spring is in the air, and that makes up for it.

by Joel Spolsky

Human Task Switches Considered Harmful

When you’re managing a team of programmers, one of the first things you have to learn to get right is task allocation. That’s just a five-dollar word for giving people things to do. It’s known colloquially as “file dumping” in Hebrew (because you dump files in peoples’ laps). And how you decide which files to dump in which laps is one of the areas where you can get incredible productivity benefits if you do it right. Do it wrong, and you can create one of those gnarly situations where nobody gets anything accomplished and everybody complains that “nothing ever gets done around here.”

Since this site is for programmers, I’m going to warm up your brains a little bit with a programming problem. 

Suppose you have two separate computations to perform, A and B. Each computation requires 10 seconds of CPU time. You have one CPU which, for the sake of this problem, doesn’t have anything else in the queue.

On our CPU, multitasking is optional. So you can either do these computations one after the other…

Sequential Processing

Computation A Computation B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

… or, you can multitask. If you multitask, on this particular CPU, tasks run for 1 second at a time, and a task switch takes no time at all.

Multitasking

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Which would you rather do?  Most people’s gut reaction is that multitasking is better. In both cases, you have to wait 20 seconds to get both of your answers. But think about how long it takes to get the results to each computation.

In both cases, the results of Computation B (shown in blue) take 20 seconds to arrive. But look at Computation A. With multitasking, its results take 19 seconds to arrive… yet with sequential processing they are ready in only 10 seconds.

In other words, in this nice contrived example, the average time per computation is lower (15 seconds rather than 19.5 seconds) when you do sequential processing rather than multitasking. (Actually, it’s not such a contrived example — it’s based on a real problem Jared had to solve at work).

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 seconds 15
Multitasking 19 seconds 20 seconds 19.5

Earlier I said that “a task switch takes no time at all.” Actually, on real CPUs, a task switch takes a little bit of time… basically enough time to save out the state of the CPU registers and load the CPU registers for the other task. Realistically, this is as close to negligible as possible. But to make life interesting, let’s imagine that task switches take half a second. Now things look even worse:

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 + 1 task switch =
20.5 seconds		 15.25
Multitasking 19 + 18 task switches =
28 seconds		 20 + 19 task switches = 29.5 seconds 28.75

Now … just humor me, I know this is silly … what if task switches take a whole minute?

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 + 1 task switch =
80 seconds		 45 seconds
Multitasking 19 + 18 task switches =
1099 seconds		 20 + 19 task switches = 1160 seconds almost 19 minutes!

The longer a task switch takes, the worse the multitasking penalty.

That, in and of itself, is not so earth shaking, is it? Pretty soon I’m going to be getting irate email from morons accusing me of being “against” multitasking. “Do you want to go back to the days of DOS when you had to exit WordPerfect to run 1-2-3?” they will ask me.

But that’s not my point. I just want you to agree with me that in this kind of example:

a) sequential processing gets you results faster on average, and

b) the longer it takes to task switch, the bigger the penalty you pay for multitasking.

OK, back to the more interesting topic of managing humans, not CPUs. The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That’s because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming. A programmer coding at full throttle is keeping zillions of things in their head at once: everything from names of variables, data structures, important APIs, the names of utility functions that they wrote and call a lot, even the name of the subdirectory where they store their source code. If you send that programmer to Crete for a three week vacation, they will forget it all. The human brain seems to move it out of short-term RAM and swaps it out onto a backup tape where it takes forever to retrieve.

How long? Well, my software company recently dropped what we were doing (developing a software product codenamed CityDesk) to help a client with a bit of an emergency situation for three weeks. When we got back to the office, it seemed to take another three weeks to get back to full speed on CityDesk.

On the individual level — have you ever noticed that you can assign one job to one person, and they’ll do a great job, but if you assign two jobs to that person, they won’t really get anything done? They’ll either do one job well and neglect the other, or they’ll do both jobs so slowly you feel like slugs have more zip. That’s because programming tasks take so long to task switch. I feel like when I have two programming projects on my plate at once, the task switch time is something like 6 hours. In an 8-hour day, that means multitasking reduces my productivity to 2 hours per day. Pretty dismal.

As it turns out, if you give somebody two things to work on, you should be grateful if they “starve” one task and only work on one, because they’re going to get more stuff done and finish the average task sooner. In fact, the real lesson from all this is that you should never let people work on more than one thing at once. Make sure they know what it is. Good managers see their responsibility as removing obstacles so that people can focus on one thing and really get it done. When emergencies come up, think about whether you can handle it yourself before you delegate it to a programmer who is deeply submersed in a project.

by Joel Spolsky

Daily Builds Are Your Friend

In 1982, my family took delivery of the very first IBM-PC in Israel. We actually went down to the warehouse and waited while our PC was delivered from the port. Somehow, I convinced my dad to get the fully-decked out version, with two floppy disks, 128 K memory, and both a dot-matrix printer (for fast drafts) and a Brother Letter-Quality Daisy Wheel printer, which sounds exactly like a machine gun when it is operating, only louder. I think we got almost every accessory available: PC-DOS 1.0, the $75 technical reference manual with a complete source code listing of the BIOS, Macro Assembler, and the stunning IBM Monochrome display with a full 80 columns and … lower case letters! The whole thing cost about $10,000 including Israel’s then-ridiculous import taxes. Extravagant!

Now, “everybody” knew that BASIC was a children’s language that requires you to write spaghetti code and turns your brain into Camembert cheese. So we shelled out $600 for IBM Pascal, which came on three floppy diskettes. The compiler’s first pass was on the first diskette, the second pass was on the second diskette, and the linker was on the third diskette. I wrote a simple “hello, world” program and compiled it. Total time elapsed: 8 minutes.

Hmm. That’s a long time. I wrote a batch file to automate the process and shaved it down to 7 1/2 minutes. Better. But when I tried to write long programs like my stunning version of Othello which always beat me, I spent most of the time waiting for compiles. “Yep,” a professional programmer told me, “we used to keep a sit-up board in the office and do sit-ups while we were doing compiles. After a few months of programming I had killer abs.”

One day, a spiffy program called Compas Pascal appeared from Denmark, which Philippe Kahn bought and renamed Borland Turbo Pascal. Turbo Pascal was sort of shocking, since it basically did everything that IBM Pascal did, only it ran in about 33K of memory including the text editor. This was nothing short of astonishing. Even more astonishing was the fact that you could compile a small program in less than one second. It’s as if a company you had never heard of introduced a clone of the Buick LeSabre which could go 1,000,000 MPH and drive around the world on so little gasoline than an ant could drink it without getting sick.

Suddenly, I became much more productive.

That’s when I learned about the concept of the REP loop. REP stands for “Read, Eval, Print”, and it describes what a lisp interpreter does for a living: it “reads” your input, evaluates it, and prints the result. An example of the REP loop is shown below: I type something, and the lisp interpreter reads it, evaluates it, and prints the result.

picture-reploop:

On a slightly larger scale, when you’re writing code, you are in a macro-version of the REP loop called the Edit-Compile-Test loop. You edit your code, compile it, test it, and see how well it works.

A crucial observation here is that you have to run through the loop again and again to write a program, and so it follows that the faster the Edit-Compile-Test loop, the more productive you will be, down to a natural limit of instantaneous compiles. That’s the formal, computer-science-y reason that computer programmers want really fast hardware and compiler developers will do anything they can to get super-fast Edit-Compile-Test loops. Visual Basic does it by parsing and lex-ing each line as you type it, so that the final compile can be super-quick. Visual C++ does it by providing incremental compiles, precompiled headers, and incremental linking.

But as soon as you start working on a larger team with multiple developers and testers, you encounter the same loop again, writ larger (it’s fractal, dude!). A tester finds a bug in the code, and reports the bug. The programmer fixes the bug. How long does it take before the tester gets the fixed version of the code? In some development organizations, this Report-Fix-Retest loop can take a couple of weeks, which means the whole organization is running unproductively. To keep the whole development process running smoothly, you need to focus on getting the Report-Fix-Retest loop tightened.

One good way to do this is with daily builds. A daily build is an automatic, daily, complete build of the entire source tree.

Automatic – because you set up the code to be compiled at a fixed time every day, using cron jobs (on UNIX) or the scheduler service (on Windows).

Daily – or even more often. It’s tempting to do continuous builds, but you probably can’t, because of source control issues which I’ll talk about in a minute.

Complete – chances are, your code has multiple versions. Multiple language versions, multiple operating systems, or a high-end/low-end version. The daily build needs to build all of them. And it needs to build every file from scratch, not relying on the compiler’s possibly imperfect incremental rebuild capabilities.

Here are some of the many benefits of daily builds:

  1. When a bug is fixed, testers get the new version quickly and can retest to see if the bug was really fixed.
  2. Developers can feel more secure that a change they made isn’t going to break any of the 1024 versions of the system that get produced, without actually having an OS/2 box on their desk to test on.
  3. Developers who check in their changes right before the scheduled daily build know that they aren’t going to hose everybody else by checking in something which “breaks the build” — that is, something that causes nobody to be able to compile. This is the equivalent of the Blue Screen of Death for an entire programming team, and happens a lot when a programmer forgets to add a new file they created to the repository. The build runs fine on their machines, but when anyone else checks out, they get linker errors and are stopped cold from doing any work.
  4. Outside groups like marketing, beta customer sites, and so forth who need to use the immature product can pick a build that is known to be fairly stable and keep using it for a while.
  5. By maintaining an archive of all daily builds, when you discover a really strange, new bug and you have no idea what’s causing it, you can use binary search on the historical archive to pinpoint when the bug first appeared in the code. Combined with good source control, you can probably track down which check-in caused the problem.
  6. When a tester reports a problem that the programmer thinks is fixed, the tester can say which build they saw the problem in. Then the programmer looks at when he checked in the fix and figure out whether it’s really fixed.

Here’s how to do them. You need a daily build server, which will probably be the fastest computer you can get your hands on. Write a script which checks out a complete copy of the current source code from the repository (you are using source control, aren’t you?) and then builds, from scratch, every version of the code that you ship. If you have an installer or setup program, build that too. Everything you ship to customers should be produced by the daily build process. Put each build in its own directory, coded by date. Run your script at a fixed time every day.

  • It’s crucial that everything it takes to make a final build is done by the daily build script, from checking out the code up to and including putting the bits up on a web server in the right place for the public to download (although during the development process, this will be a test server, of course). That’s the only way to insure that there is nothing about the build process that is only “documented” in one person’s head. You never get into a situation where you can’t release a product because only Shaniqua knows how to create the installer, and she was hit by a bus. On the Juno team, the only thing you needed to know to create a full build from scratch was where the build server was, and how to double-click on its “Daily Build” icon.
  • There is nothing worse for your sanity then when you are trying to ship the code, and there’s one tiny bug, so you fix that one tiny bug right on the daily build server and ship it. As a golden rule, you should only ship code that has been produced by a full, clean daily build that started from a complete checkout.
  • Set your compilers to maximum warning level (-W4 in Microsoft’s world; -Wall in gcc land) and set them to stop if they encounter even the smallest warning.
  • If a daily build is broken, you run the risk of stopping the whole team. Stop everything and keep rebuilding until it’s fixed. Some days, you may have multiple daily builds.
  • Your daily build script should report failures, via email, to the whole development team. It’s not too hard to grep the logs for “error” or “warning” and include that in the email, too. The script can also append status reports to an HTML page visible to everyone so programmers and testers can quickly determine which builds were successful.
  • One rule we followed on the Microsoft Excel team, to great effect, was that whoever broke the build became responsible for babysitting builds until somebody else broke it. In addition to serving as a clever incentive to keep the build working, it rotated almost everybody through the job of buildmaster so everybody learned about how builds are produced.
  • If your team works in one time zone, a good time to do builds is at lunchtime. That way everybody checks in their latest code right before lunch, the build runs while they’re eating, and when they get back, if the build is broken, everybody is around to fix it. As soon as the build is working everybody can check out the latest version without fear that they will be hosed due to a broken build.
  • If your team is working in two time zones, schedule the daily build so that the people in one time zone don’t hose the people in the other time zone. On the Juno team, the New York people would check things in at 7 PM New York time and go home. If they broke the build, the Hyderabad, India team would get into work (at about 8 PM New York Time) and be completely stuck for a whole day. We started doing two daily builds, about an hour before each team went home, and completely solved that problem.

For Further Reading:

  • Some discussion on tools for daily builds
  • Making daily builds is important enough that it’s one of the 12 steps to better code.
  • There’s a lot of interesting stuff about the builds made (weekly) by the Windows NT team in G. Pascal Zachary’s book Showstopper.
  • Steve McConnell writes about daily builds here.
by Joel Spolsky

Big Macs vs. The Naked Chef

Mystery: why is it that some of the biggest IT consulting companies in the world do the worst work?

Why is it that the cool upstart consulting companies start out with a string of spectacular successes, meteoric growth, and rapidly degenerate into mediocrity?

I’ve been thinking about this, and thinking about how Fog Creek Software (my own company) should grow. And the best lessons I can find come from McDonald’s. Yes, I mean the awful hamburger chain.

The secret of Big Macs is that they’re not very good, but every one is not very good in exactly the same way. If you’re willing to live with not-very-goodness, you can have a Big Mac with absolutely no chance of being surprised in the slightest.

The other secret of Big Macs is that you can have an IQ that hovers somewhere between “idiot” and “moron” (to use the technical terms) and you’ll still be able to produce Big Macs that are exactly as unsurprising as all the other Big Macs in the world. That’s because McDonald’s real secret sauce is its huge operations manual, describing in stunning detail the exact procedure that every franchisee must follow in creating a Big Mac. If a Big Mac hamburger is fried for 37 seconds in Anchorage, Alaska, it will be fried for 37 seconds in Singapore – not 36, not 38. To make a Big Mac you just follow the damn rules.

The rules have been carefully designed by reasonably intelligent people (back at McDonald’s Hamburger University) so that dumdums can follow them just as well as smart people. In fact the rules include all kinds of failsafes, like bells that go off if you keep the fries in the oil too long, which were created to compensate for more than a little human frailty. There are stopwatches and timing systems everywhere. There is a system to make sure that the janitor checks if the bathrooms are clean every half hour. (Hint: they’re not.)

The system basically assumes that everybody will make a bunch of mistakes, but the burgers that come out will be, um, consistent, and you’ll always be asked if you want fries with that.

Just for the sake of amusement, let’s compare a McDonald’s cook, who is following a set of rules exactly and doesn’t know anything about food, to a genius like The Naked Chef, the British cutie Jamie Oliver. (If you chose to leave this site now and follow that link to watch the MTV-like videos of The Naked Chef making basil aioli, you have my blessing. Go in good health.) Anyway, comparing McDonald’s to a gourmet chef is completely absurd, but please suspend disbelief for a moment, because there’s something to be learned here.

Now, the Naked Chef doesn’t follow no stinkin’ Operations Manual. He doesn’t measure anything. While he’s cooking, you see a flurry of food tossed around willy-nilly. “We’ll just put a bit of extra rosemary in there, that won’t hurt, and give it a good old shake,” he says. ” Mash it up. Perfect. Just chuck it all over the place.” (Yes, it really looks like he’s just chucking it all over the place. Sorry, but if I tried to chuck it all over the place, it wouldn’t work.) It takes about 14 seconds and he’s basically improvised a complete gourmet meal with roasted slashed fillet of sea-bass stuffed with herbs, baked on mushroom potatoes with a salsa-verde. Yum.

Well, I think it’s pretty obvious that The Naked Chef’s food is better than you get at McDonald’s. Even if it sounds like a stupid question, it’s worth a minute to ask why. It’s not such a stupid question. Why can’t a big company with zillions of resources, incredible scale, access to the best food designers money can buy, and infinite cash flow produce a nice meal?

Imagine that The Naked Chef gets bored doing “telly” and opens a restaurant. Of course, he’s a brilliant chef, the food would be incredible, so the place is hopping with customers and shockingly profitable.

When you have a shockingly profitable restaurant, you quickly realize that even if you fill up every night, and even if you charge $19 for an appetizer and $3.95 for a coke, your profits reach a natural limit, because one chef can only make so much food. So you hire another chef, and maybe open some more branches, maybe in other cities.

Now a problem starts to develop: what we in the technical fields call the scalability problem. When you try to clone a restaurant, you have to decide between hiring another great chef of your caliber (in which case, that chef will probably want and expect to keep most of the extra profits that he created, so why bother), or else you’ll hire a cheaper, younger chef who’s not quite as good, but pretty soon your patrons will figure that out and they won’t go to the clone restaurant.

The common way of dealing with the scalability problem is to hire cheap chefs who don’t know anything, and give them such precise rules about how to create every dish that they “can’t” screw it up. Just follow these here rules, and you’ll make great gourmet food!

Problem: it doesn’t work exactly right. There are a million things that a good chef does that have to do with improvisation. A good chef sees some awesome mangos in the farmer’s market and improvises a mango-cilantro salsa for the fish of the day. A good chef deals with a temporary shortage of potatoes by creating some taro chip thing. An automaton chef who is merely following instructions might be able to produce a given dish when everything is working perfectly, but without real talent and skill, will not be able to improvise, which is why you never see jicama at McDonald’s.

McDonald’s requires a very particular variety of potato, which they grow all over the world, and which they pre-cut and freeze in massive quantities to survive shortages. The precutting and freezing means that the french-fries are not as good as they could be, but they are certainly consistent and require no chef-skills. In fact, McDonald’s does hundreds of things to make sure that their product can be produced with consistent quality, by any moron you can get in the kitchen, even if the quality is “a bit” lower.

Summary, so far:

  1. Some things need talent to do really well.
  2. It’s hard to scale talent.
  3. One way people try to scale talent is by having the talent create rules for the untalented to follow.
  4. The quality of the resulting product is very low.

You can see the exact same story playing out in IT consulting. How many times have you heard this story?

Mike was unhappy. He had hired a huge company of IT consultants to build The System. The IT consultants he hired were incompetents who kept talking about “The Methodology” and who spent millions of dollars and had failed to produce a single thing.

Luckily, Mike found a youthful programmer who was really smart and talented. The youthful programmer built his whole system in one day for $20 and pizza. Mike was overjoyed. He recommended the youthful programmer to all his friends.

Youthful Programmer starts raking in the money. Soon, he has more work than he can handle, so he hires a bunch of people to help him. The good people want too many stock options, so he decides to hire even younger programmers right out of college and “train them” with a 6 week course.

The trouble is that the “training” doesn’t really produce consistent results, so Youthful Programmer starts creating rules and procedures that are meant to make more consistent results. Over the years, the rule book grows and grows. Soon it’s a six-volume manual called The Methodology.

After a few dozen years, Youthful Programmer is now a Huge Incompetent IT Consultant with a capital-M-methodology and a lot of people who blindly obey the Methodology, even when it doesn’t seem to be working, because they have no bloody idea whatsoever what else to do, and they’re not really talented programmers — they’re just well-meaning Poli Sci majors who attended the six-week course.

And Newly Huge Incompetent IT Consultant starts messing up. Their customers are unhappy. And another upstart talented programmer comes and takes away all their business, and the cycle begins anew.

I don’t need to name names, here, this cycle has happened a dozen times. All the IT service companies get greedy and try to grow faster than they can find talented people, and they grow layers upon layers of rules and procedures which help produce “consistent,” if not very brilliant work.

But the rules and procedures only work when nothing goes wrong. Various “data-backed Web site” consulting companies sprouted up in the last couple of years and filled their ranks by teaching rank amateurs the fourteen things you need to know to create a data-backed Web site (“here’s a select statement, kid, build a Web site”). But now that dotcoms are imploding and there’s suddenly demand for high-end GUI programming, C++ skills, and real computer science, the kids who only have select statements in their arsenal just have too steep a learning curve and can’t catch up. But they still keep trying, following the rules in chapter 17 about normalizing databases, which mysteriously don’t apply to The New World. The brilliant founders of these companies could certainly adjust to the new world: they are talented computer scientists who can learn anything, but the company they built can’t adjust because it has substituted a rulebook for talent, and rulebooks don’t adjust to new times.

What’s the moral of the story? Beware of Methodologies. They are a great way to bring everyone up to a dismal, but passable, level of performance, but at the same time, they are aggravating to more talented people who chafe at the restrictions that are placed on them. It’s pretty obvious to me that a talented chef is not going to be happy making burgers at McDonald’s, precisely because of McDonald’s rules. So why do IT consultants brag so much about their methodologies? (Beats me.)

What does this mean for Fog Creek? Well, our goal has never been to become a huge consulting company. We started out doing consulting as a means to an end — the long-term goal was to be a software company that is always profitable, and we achieved that by doing some consulting work to supplement our software income. After a couple of years in business our software revenues grew to the point where consulting was just a low-margin distraction, so now we only do consulting engagements that directly support our software. Software, as you know, scales incredibly well. When one new person buys FogBUGZ, we make more money without spending any more money.

More important is our obsession with hiring the best… we are perfectly happy to stay small if we can’t find enough good people (although with six weeks annual vacation, finding people doesn’t seem to pose a problem). And we refuse to grow until the people we already hired have learned enough to become teachers and mentors of the new crowd.

by Joel Spolsky

Painless Bug Tracking

TRS-80 Level-I BASIC could only store two string variables, A$ and B$. Similarly, I was born with only two bug-storing-slots in my brain. At any given time, I can only remember two bugs. If you ask me to remember three, one of them will fall on the floor and get swept under the bed with the dust bunnies, who will eat it.

Keeping a database of bugs is one of the hallmarks of a good software team. I never cease to be amazed at how few teams are actually doing this. One of the biggest incorrect facts that programmers consistently seem to believe is that they can remember all their bugs or keep them on post-it notes.

If I can bend your ear a moment, I’d like to explain a pretty painless way to do bug tracking, in the spirit of my previous articles on painless schedules and painless specs.

First of all, you need a real database. On teams of 2 people writing a little bit of code over the course of a long weekend, it’s probably OK to use a text file as the database. Anything larger, and you’re going to need a real bug tracking database. There are zillions of bug tracking databases you can buy. (Blatant self-promotion: the bug tracker we wrote at Fog Creek Software, called FogBugz, is web based, pretty easy to use, and quite powerful, if I may say so myself.)

Let’s follow a bug around, for the purpose of illustration, from the moment it’s born until someone finally puts it out of its misery. We’ll follow the famous Bug 1203. Here’s what the bug database shows for that bug:

ID   1203
Project   Bee Flogger 2.0 
Area   FTP Client
Title   Uploading file causes FTP server to dump core
Assigned To   CLOSED
Status   CLOSED (RESOLVED – FIXED)
Priority   2 – Must Fix
Fix For   2.0 Alpha
Version   Build 2019
Computer   Jill’s iMac, Mac OS 9.0, 128M RAM, 1024×768 millions of colors
Description   11/1/2000 Opened by Jill the Very, Very Good Tester

* Start Bee Flogger
* Create an unnamed document simply containing the letter “a”
* Click on the FTP button on the toolbar
* Try to ftp to your server

BUG: Observe; the ftp server is no longer responding. Indeed ps -augx shows that it is not even running and there is a core dump in /.

EXPECTED: No crash

11/1/2000 Assigned to Willie the Lead Developer by Jill the Very, Very Good Tester

11/2/2000 (Yesterday) RESOLVED – WON’T FIX by Willie the Lead Developer

Not our code, Jill, that’s just proftpd which comes with Linux.

11/2/2000 (Yesterday) Reactivated (assigned to Willie the Lead Developer) by Jill the Very, Very Good Tester

That doesn’t sound right. I’ve never been able to crash proftpd when I connect with a normal ftp client. Our code crashes it every single time. Ftp servers don’t just “crash”.

11/3/2000 (Today) Assigned to Mikey the Programmer by Willie the Lead Developer

Mikey, can you look at this? Maybe your client code is doing something wrong.

11/3/2000 (Today) RESOLVED – FIXED by Mikey the Programmer

I think I was passing the user name instead of the password or something…

11/3/2000 (Today) Reactivated (assigned to Mikey the Programmer) by Jill the Very, Very Good Tester

Still happens in Build 2021.

11/3/2000 (Today) Edited by Mikey the Programmer

Whoa. That’s strange. Lemme debug this.

11/3/2000 (Today) Edited by Mikey the Programmer

I’m thinking it might be MikeyStrCpy()…

11/3/2000 (Today) RESOLVED – FIXED by Mikey the Programmer

Ahhh!
FIXED!

11/3/2000 (Today) Closed by Jill the Very, Very Good Tester

Appears fixed in build 2022, so I’ll go ahead and close this.

Here’s what happened.

Mikey the Programmer is hacking away on the new FTP client feature of his groovy Macintosh software. At some point, because he’s feeling frisky, he writes his own string-copy function. That’ll teach them pesky reusability police! Bwa ha ha!

Bad things happen when you don’t reuse code, Mikey. And today, the bad thing that happened was that Mikey forgot to null-terminate the copied string. But he never noticed the problem because most of the time he happened to be copying strings into pre-zeroed memory.

Later that week, Jill the Very, Very Good Tester is banging away at the code, rolling her forehead back and forth on the keyboard or some equally cruel test.  (Incidentally, most good testers are named Jill, or some variation like Gillian.) Suddenly something very strange happens: the ftp daemon she was testing against crashed. Yes, I know it’s a Linux machine and Linux machines never crash (no snorting sounds from the slashdot crowd, please) but this dang thing crashed. And she wasn’t even touching the server, she was just FTPing files to it using Mikey’s Mac code.

Now, Jill is a very, very good tester, so she’s kept a careful log of what she was doing (the precise pitch and yaw of her head as she rolled it on the keyboard is in her lab book, for example). She reboots everything, starts with a clean machine, repeats the steps, and — Lo and Behold — it happens again! The Linux ftp daemon crashed again! That’s twice in one day, now! Take that, Linus.

Jill squints at the list of repro steps. There are about 20 steps. Some of them don’t seem related. After a bit of experimentation, Jill is able to whittle the problem down to four steps that always cause the same behavior. Now she’s ready to file a bug.

Jill enters the new bug in the bug tracking database. Now, just the act of entering a bug requires some discipline: there are good bug reports and bad bug reports.

Three Parts To Every Good Bug Report

And the Lord spake, saying, “First shalt thou take out the Holy Pin. Then, shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shalt be three. Four shalt thou not count, nor either count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then lobbest thou thy Holy Hand Grenade of Antioch towards thou foe, who being naughty in my sight, shall snuff it.”

— Monty Python and the Holy Grail

It’s pretty easy to remember the rule for a good bug report. Every good bug report needs exactly three things.

  1. Steps to reproduce,
  2. What you expected to see, and
  3. What you saw instead.

Seems easy, right? Maybe not. As a programmer, people regularly assign me bugs where they left out one piece or another.

If you don’t tell me how to repro the bug, I probably will have no idea what you are talking about. “The program crashed and left a smelly turd-like object on the desk.” That’s nice, honey. I can’t do anything about it unless you tell me what you were doing. Now, I admit that there are two cases where it’s hard to get exact steps to repro. Sometimes you just don’t remember, or you’re just transcribing a bug from “the field.” (By the way, why do they call it “the field”? Is it, like, a field of rye or something? Anyway…) The other time it’s OK not to have repro steps is when the bug happens sometimes but not all the time, but you should still provide repro steps, with a little annotation that says that it doesn’t happen too often. In these cases, it’s going to be really hard to find the bug, but we can try.

If you don’t specify what you expected to see, I may not understand why this is a bug. The splash screen has blood on it. So what? I cut my fingers when I was coding it. What did you expect? Ah, you say that the spec required no blood! Now I understand why you consider this a bug.

Part three. What you saw instead. If you don’t tell me this, I don’t know what the bug is. That one is kind of obvious.

Back To Our Story

Anyhoo. Jill enters the bug. In a good bug tracking system it gets automatically assigned to the lead developer for that project. And therein lies the second concept: every bug needs to be assigned to exactly one person at all times, until it is closed. A bug is like a hot potato: when it’s assigned to you, you are responsible to resolve it, somehow, or assign it to someone else.

Willie, the lead developer, looks at the bug, decides it’s probably something to do with the ftp server, and resolves it as “won’t fix.” After all, they didn’t write the ftp server.

When a bug is resolved, it gets assigned back to the person who opened it. This is a crucial point. It does not go away just because a programmer thinks it should. The golden rule is that only the person who opened the bug can close the bug. The programmer can resolve the bug, meaning, “hey, I think this is done,” but to actually close the bug and get it off the books, the original person who opened it needs to confirm that it was actually fixed or agree that it shouldn’t be fixed for some reason.

Jill gets an email telling her that the bug is back in her court. She looks at it and reads Willie the Lead Developer’s comments. Something doesn’t sound right. People have been using this ftp server for years and it doesn’t crash. It only crashes when you use Mikey’s code. So Jill reactivates the bug explaining her position, and the bug goes back to Willie. This time Willie assigns the bug to Mikey to fix.

Mikey studies the bug, thinks long and hard, and completely misdiagnoses the bug. He fixes some altogether different bug, and then resolves the one Jill opened.

The bug is back to Jill, this time marked “RESOLVED-FIXED”. Jill tries her repro steps with the latest build, and, lo and behold, the Linux server crashes. She reactivates the bug again and assigns it straight back to Mikey.

Mikey is perplexed, but he finally tracks down the source of the bug. (Know what it is yet? I’ll leave it as an exercise to the reader. I’ve given you enough clues!) He fixes it, tests it, and — Eureka! The repro case no longer crashes the ftp server. Once again, he resolves it as FIXED. Jill also tries the repro steps, discovers that the bug is good ‘n’ fixed, and closes it out.

Top Ten Tips for Bug Tracking

  1. A good tester will always try to reduce the repro steps to the minimal steps to reproduce; this is extremely helpful for the programmer who has to find the bug.
  2. Remember that the only person who can close a bug is the person who opened it in the first place. Anyone can resolve it, but only the person who saw the bug can really be sure that what they saw is fixed.
  3. There are many ways to resolve a bug. FogBUGZ allows you to resolve a bug as fixed, won’t fix, postponed, not repro, duplicate, or by design.
  4. Not Repro means that nobody could ever reproduce the bug. Programmers often use this when the bug report is missing the repro steps.
  5. You’ll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn’t have to retest the bug on a version of the software where it wasn’t even supposed to be fixed.
  6. If you’re a programmer, and you’re having trouble getting testers to use the bug database, just don’t accept bug reports by any other method. If your testers are used to sending you email with bug reports, just bounce the emails back to them with a brief message: “please put this in the bug database. I can’t keep track of emails.”
  7. If you’re a tester, and you’re having trouble getting programmers to use the bug database, just don’t tell them about bugs – put them in the database and let the database email them.
  8. If you’re a programmer, and only some of your colleagues use the bug database, just start assigning them bugs in the database. Eventually they’ll get the hint.
  9. If you’re a manager, and nobody seems to be using the bug database that you installed at great expense, start assigning new features to people using bugs. A bug database is also a great “unimplemented feature” database, too.
  10. Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It’s very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs “formally” is too much work, people will go around the bug database.

If you are developing code, even on a team of one, without an organized database listing all known bugs in the code, you are simply going to ship low quality code. On good software teams, not only is the bug database used universally, but people get into the habit of using the bug database to make their own “to-do” lists, they set their default page in their web browser to the list of bugs assigned to them, and they start wishing that they could assign bugs to the office manager to stock more Mountain Dew.

by Joel Spolsky

Painless Functional Specifications – Part 4: Tips

OK, we’ve talked about  why you need a spec, what a spec has in it, and who should write them. In this fourth and final part of the series I’ll share some of my advice for writing good specs.

The biggest complaint you’ll hear from teams that do write specs is that “nobody reads them.” When nobody reads specs, the people who write them tend to get a little bit cynical. It’s like the old Dilbert cartoon in which engineers use stacks of 4-inch thick specs to build extensions to their cubicles. At your typical big, bureaucratic company, everybody spends months and months writing boring specs. Once the spec is done, it goes up on the shelf, never to be taken down again, and the product is implemented from scratch without any regard to what the spec said, because nobody read the spec, because it was so dang mind-numbing. The very process of writing the spec might have been a good exercise, because it forced everyone, at least, to think over the issues. But the fact that the spec was shelved (unread and unloved) when it was completed makes people feel like it was all a bunch of work for naught.

Also, if your spec never gets read, you get a lot of arguments when the finished product is delivered. Somebody (management, marketing, or a customer) says: “wait a minute! You promised me that there would be a Clam Steamer! Where’s the clam steamer?” And the programmers say, “no, actually, if you look on the spec on chapter 3, subchapter 4, paragraph 2.3.0.1, you’ll see it says quite explicitly ‘no clam steamer.'” But that doesn’t satisfy the customer, who is always right, so the grumpy programmers have to go retrofit a clam steamer into the thing (making them even more cynical about specs). Or a manager says, “hey, all the wording on this dialog is too verbose, and there should be an advertisement at the top of every dialog box.” And the programmers say, in frustration, “but you approved the spec which precisely listed the layout and contents of every dialog box!” But of course, the manager hadn’t actually read the spec, because when he tried, his brain started seeping out through his eye sockets, and anyway, it was interfering with his Tuesday golf game.

So. Specs are good, but not if nobody reads them. As a spec-writer, you have to trick people into reading your stuff, and you should also probably make an effort not to cause any already-too-small brains to leak out through eye-sockets.

Tricking people into reading your stuff is usually just a matter of good writing. But it’s not fair of me to just say “be a good writer” and leave it at that. Here are four easy rules that you absolutely must follow to make specs that get read.

Rule 1: Be Funny

Yep, rule number one in tricking people into reading your spec is to make the experience enjoyable. Don’t tell me you weren’t born funny, I don’t buy it. Everybody has funny ideas all the time, they just self-censor them because they think that it’s “unprofessional.” Feh. Sometimes you have to break the rules.

If you read the volumes of garbage I’ve written on this web site, you’ll notice that there are a few lame attempts at being funny scattered throughout. Just four paragraphs ago I was making a gross body-fluid joke and making fun of managers for playing golf. Even though I’m not really that funny, I still try pretty hard, and even the act of flailing around trying to be funny is in itself amusing, in a sad-clown sort of way. When you’re writing a spec, an easy place to be funny is in the examples. Every time you need to tell a story about how a feature works, instead of saying:

  • The user types Ctrl+N to create a new Employee table and starts entering the names of the employees.

write something like:

  • Miss Piggy, poking at the keyboard with a eyeliner stick because her chubby little fingers are too fat to press individual keys, types Ctrl+N to create a new Boyfriend table and types in the single record “Kermit.”

If you read a lot of Dave Barry, you’ll discover that one of the easiest ways to be funny is to be specific when it’s not called for. “Scrappy pugs” are funnier than “dogs.” “Miss Piggy” is funnier than “the user”. Instead of saying “special interests,” say “left-handed avocado farmers.” Instead of saying “People who refuse to clean up after their dogs should be punished,” say that they should be “sent to prisons so lonely that the inmates have to pay spiders for sex.”

Oh, and, by the way, if you think that it’s unprofessional to be funny, then I’m sorry, but you just don’t have a sense of humor. (Don’t deny it. People without senses of humors always deny it. You can’t fool me.) And if you work in a company where people will respect you less because your specs are breezy, funny, and enjoyable to read, then go find another company to work for, because life is just too damn short to spend your daylight hours in such a stern and miserable place.

Rule 2: Writing a spec is like writing code for a brain to execute

Here’s why I think that programmers have trouble writing good specs.

When you write code, your primary audience is the compiler. Yeah, I know, people have to read code, too, but it’s generally very hard for them. For most programmers it’s hard enough to get the code into a state where the compiler reads it and correctly interprets it; worrying about making human-readable code is a luxury. Whether you write:

void print_count( FILE* a, char  *  b, int c ){
    fprintf(a, “there are %d %s\n”, c, b);}

main(){ int n; n =
10; print_count(stdout, “employees”, n) /* code
deliberately obfuscated */ }

or

printf(“there are 10 employees\n”);

you get the same output. Which is why, if you think about it, you tend to get programmers who write things like:

Assume a function AddressOf(x) which is defined as the mapping from a user x, to the RFC-822 compliant email address of that user, an ANSI string. Let us assume user A and user B, where A wants to send an email to user B. So user A initiates a new message using any (but not all) of the techniques defined elsewhere, and types AddressOf(B) in the To: editbox.

This could also have been speced as:

Miss Piggy wants to go to lunch, so she starts a new email and types Kermit’s address in the “To:” box. 

Technical note: the address must be a standard Internet address (RFC-822 compliant.)

They both “mean” the same thing, theoretically, except that the first example is impossible to understand unless you carefully decode it, and the second example is easy to understand. Programmers often try to write specs which look like dense academic papers. They think that a “correct” spec needs to be “technically” correct and then they are off the hook.

The mistake is that when you write a spec, in addition to being correct, it has to be understandable, which, in programming terms, means that it needs to be written so that the human brain can “compile” it. One of the big differences between computers and human brains is that computers are willing to sit there patiently while you define the terms that you want to use later. But humans won’t understand what you’re talking about unless you motivate it first. Humans don’t want to have to decode something, they just want to read it in order and understand it. For humans, you have to provide the big picture and then fill in the details. With computer programs, you start at the top and work your way to the bottom, with full details throughout. A computer doesn’t care if your variable names are meaningful. A human brain understands things much better if you can paint a vivid picture in their mind by telling a story, even if it’s just a fragment of a story, because our brains have evolved to understand stories.

If you show a chess board, in the middle of a real game of chess, to an experienced chess player for even a second or two, they will instantly be able to memorize the position of every piece. But if you move around a couple of pieces in nonsensical ways that couldn’t happen in normal play (for example, put some pawns on the first row, or put both black bishops on black squares), it becomes much, much harder for them to memorize the board. This is different from the way computers think. A computer program that could memorize a chess board could memorize both possible and impossible layouts with equal ease. The way the human brain works is not random access; pathways tend to be strengthened in our brains and some things are just easier to understand than other things because they are more common.

So, when you’re writing a spec, try to imagine the person you are addressing it to, and try to imagine what you’re asking them to understand at every step. Sentence by sentence, ask yourself if the person reading this sentence will understand it at a deep level, in the context of what you’ve already told them. If some members of your target audience don’t know what RFC-822 is, you either have to define it, or, at the very least, bury the mention of RFC-822 in a technical note, so that the executive-types who read the spec won’t give up and stop reading the first time they see a lot of technical jargon. 

Rule 3: Write as simply as possible

Don’t use stilted, formal language because you think it’s unprofessional to write in simple sentences. Use the simplest language you can. 

People use words like “utilize” because they think that “use” looks unprofessional. (There’s that word “unprofessional” again. Any time somebody tells you that you shouldn’t do something because it’s “unprofessional,” you know that they’ve run out of real arguments.) In fact I think that many people think that clear writing means that something is wrong.

Break things down to short sentences. If you’re having trouble writing a sentence clearly, break it into two or three shorter sentences. 

Avoid walls of text: entire pages with just text. People get scared and don’t read them. When was the last time you noticed a popular magazine or newspaper with entire pages of text? Magazines will go so far as to take a quote from the article and print it, in the middle of the page, in a giant font, just to avoid the appearance of a full page of text. Use numbered or bulleted lists, pictures, charts, tables, and lots of whitespace so that the reading “looks” fluffier.

              

“Magazines will go so far as to take a quote from the article and print it, in the middle of the page, in a giant font, just to avoid the appearance of a full page of text.”

 

Nothing improves a spec more than lots and lots of screenshots. A picture can be worth a thousand words. Anyone who writes specs for Windows software should invest in a copy of Visual Basic, and learn to use it at least well enough to create mockups of the screens. (For the Mac, use REAL Basic; for Web pages, use Front Page or Dreamweaver). Then capture these screenshots (Ctrl+PrtSc) and paste them into your spec.

Rule 4: Review and reread several times

Um, well, I was originally planning to have a lengthy exegesis of this rule here, but this rule is just too simple and obvious. Review and reread your spec several times, OK? When you find a sentence that isn’t super easy to understand, rewrite it.

I’ve saved so much time by not explaining Rule 4 that I’m going to add another rule.

Rule 5: Templates considered harmful

Avoid the temptation to make a standard template for specs. At first you might just think that it’s important that “every spec look the same.” Hint: it’s not. What difference does it make? Does every book on your bookshelf at home look exactly the same? Would you want them to?

Worse, if you have templates, what tends to happen is that you add a bunch of sections to the template for things that you think are important for every feature. Example: Big Bill decrees that from here on forward, every Microsquish product shall have an Internet component. So the spec template now has a section that says “Internet Component.” Whenever somebody writes a spec, no matter how trivial, they have to fill in that section that says “Internet Component”, even if they’re just creating the spec for the Microsquish Keyboard. (And you wondered why those useless Internet shopping buttons started cropping up like mushrooms on keyboards).

As these sections accumulate, the template gets pretty large. (Here is an example of a very, very bad template for specifications. Who needs a bibliography in a spec, for heaven’s sake? Or a glossary?) The trouble with such a large template is that it scares people away from writing specs because it looks like such a daunting task.

A spec is a document that you want people to read. In that way, it is no different than an essay in The New Yorker or a college paper. Have you ever heard of a professor passing out templates for students to write their college papers? Have you ever read two good essays that could be fit into a template? Just drop the idea.