A review of Dreaming in Code, by Scott Rosenberg.

Eyes work using a page fault mechanism. They’re so good at it that you don’t even notice.

You can only see at a high-resolution in a fairly small area, and even that has a big fat blind spot right exactly in the middle, but you still walk around thinking you have a ultra-high resolution panoramic view of everything. Why? Because your eyes move really fast, and, under ordinary circumstances, they are happy to jump instantly to wherever you need them to jump to. And your mind provides this really complete abstraction, providing you with the illusion of complete vision when all you really have is a very small area of high res vision, a large area of extremely low-res vision, and the ability to page-fault-in anything you want to see—so quickly that you walk around all day thinking you have the whole picture projected internally in a little theatre in your brain.

This is really, really useful, and lots of other things work this way, too. Your ears are good at tuning in important parts of conversations. Your fingers reach around and touch anything they need to, whether it’s a fine merino wool sweater or the inside of your nose, giving you a full picture of what everything feels like. When you dream, your mind asks all kinds of questions that it’s used to asking the senses (what’s that? Look over there!) but your senses are temporarily turned off (you are, after all, asleep), so they get back sort-of random answers, which you combine into a funny story in your brain called a dream. And then when you try to recount the dream to your boyfriend in the morning, even though it seemed totally, completely realistic, you suddenly realize that you don’t know what happened, actually, so you have to make shit up. If you had stayed asleep for another minute or two your brain would have asked your senses what kind of mammal was swimming with you in the rose bush, and gotten back some retarded, random answer (a platypus!), but you woke up, so until you tried to tell the story, you didn’t even realize that you needed to know what was in the rose bushes with you to make the story coherent to your partner. Which it never is. So please don’t tell me about your dreams.

One of the unfortunate side effects is that your mind gets into a bad habit of overestimating how clearly it understands things. It always thinks it has The Big Picture even when it doesn’t.

This is a particularly dangerous trap when it comes to software development. You get some big picture idea in your head for what you want to do, and it all seems so crystal clear that it doesn’t even seem like you need to design anything. You can just dive in and start implementing your vision.

Say, for example, that your vision is to rebuild an old DOS personal information manager, which was really really great but totally unappreciated. It seems easy. Everything about how the whole thing works seems so obvious, you don’t even try to design the thing… you just hire a bunch of programmers and start banging out code.

Now you’ve made two mistakes.

Number one, you fell for that old overconfidence trick of your mind. “Oh, yeah, we totally know how to do this! It’s all totally clear to us. No need to spec it out. Just write the code.”

Number two, you hired programmers before you designed the thing. Because the only thing harder than trying to design software is trying to design software as a team.

I can’t tell you how many times I’ve been in a meeting with even one or two other programmers, trying to figure out how something should work, and we’re just not getting anywhere. So I go off in my office and take out a piece of paper and figure it out. The very act of interacting with a second person was keeping me from concentrating enough to design the dang feature.

What kills me is the teams who get into the bad habit of holding meetings every time they need to figure out how something is going to work. Did you ever try to write poetry in a committee meeting? It’s like a bunch of fat construction guys trying to write an opera while sitting on the couch watching Baywatch. The more fat construction guys you add to the couch, the less likely you are to get opera out of it.

At least turn off the TV!

Now, it would be shockingly presumptuous of me to try to guess what happened on the Chandler team, and why it’s taken them millions of dollars and several years to get to where they are now, which is, they have a pretty buggy and incomplete calendar application that’s not very impressive compared to the 58 me-too Web 2.0 calendars that came out last year, each of which was developed by two college kids in their spare time, one of whom really mostly just drew mascots.

Chandler doesn’t even have a mascot!

Like I say, I can’t presume to know what went wrong. Maybe nothing. Maybe they feel like they’re right on track. Scott Rosenberg’s excellent new book, which was supposed to be a Soul of a New Machine for the hottest open source startup of the decade, ends up, in frustration, with Scott cutting the story short because Chandler 1.0 was just not going to happen any time soon (and presumably Rosenberg couldn’t run the risk that we wouldn’t be using books at all by the time it shipped, opting instead to absorb knowledge by taking a pill).

Still, it’s a great look at one particular type of software project: the kind that ends up spinning and spinning its wheels without really going anywhere because the vision was too grand and the details were a little short. Near as I can tell, Chandler’s original vision was pretty much just to be “revolutionary.” Well, I don’t know about you, but I can’t code “revolutionary.” I need more details to write code. Whenever the spec describes the product in terms of adjectives (“it will be extremely cool”) rather than specifics (“it will have brushed-aluminum title bars and all the icons will be reflected a little bit, as if placed on a grand piano”) you know you’re in trouble.

The only concrete design ideas, as far as I could tell from Rosenberg’s book, were “peer-to-peer,” “no silos,” and “natural language date interpretation.” This may be the a limitation of the book, but the initial design sure seemed to be extremely vague.

“Peer-to-peer” was the raison-d’être of Chandler… why should you have to buy Microsoft Exchange Server to coordinate schedules? It turned out that peer-to-peer synchronization was too hard, or something, and this feature was cut. Now there’s a server called Cosmo.

“No Silos” was supposed to mean that instead of having your email in one silo, and your calendar in another silo, and your reminder notes in a third, there would just be a single unified silo holding everything.

As soon as you start asking questions about “No Silos,” you realize it’s not going to work. Do you put your email on the calendar? Where? On the day when it arrived? So now I have 200 Viagra ads on Friday obscuring the one really important shareholder meeting?

Eventually “No Silos” got designed into this idea of stamps, so, for example, you could “stamp” any document or note or calendar item with an email stamp and suddenly that item could be mailed to anyone. Guess what? That feature has been in Microsoft Office for the last decade or so. They finally took it out in Office 2007 because nobody cared. There are too many easy ways to email people things.

Indeed, I think the idea of “No Silos” is most appealing to architecture astronauts, the people who look at subclasses and see abstract base classes, and who love to move functionality from the subclass into the base class for no good reason other than architectural aesthetics. This is usually a terrible user interface design technique. The way you make users understand your program model is with metaphors. When you make things look, feel, and most importantly, behave like things in the real world, users are more likely to figure out how to use the program, and the app will be easier to use. When you try to combine two very dramatically different real-world items (email and appointments) into the same kind of thing in the user interface, usability suffers because there’s no longer a real-world metaphor that applies.

The other cool thing that Kapor kept telling everyone who would listen is that Agenda would let you type things like “Next Tuesday” and magically you’d get an appointment for next Tuesday. This is slicker than heck, but every half-decent calendar program for the last decade has done this. Not revolutionary.

The Chandler team also overestimated how much help they would get from volunteers. Open source doesn’t quite work like that. It’s really good at implementing copycat features, because there’s a spec to work from: the implementation you’re copying. It’s really good at Itch Scratching features. I need a command line argument for EBCDIC, so I’ll add it and send in the code. But when you have an app that doesn’t do anything yet, nobody finds it itchy. They’re not using it. So you don’t get volunteers. Almost everyone on the Chandler dev team got paid.

Again I must forcefully apologize to the Chandler team if Rosenberg missed the point somehow, or if he gave a completely incorrect impression of what was really holding up progress, and my bias—to blame these kinds of debacles on a failure to design—is showing.

All that said, one good thing did come out of the project: a fascinating book in the style of Soul of a New Machine or Showstopper about a software development project that failed to converge. Highly recommended.