When we started selling the Aardvark DVDs on our website, we didn’t think it would be so hard to pack and ship them. First of all, we didn’t expect there would be so many, and we conveniently forgot some of the little problems like filling out customs forms which turned out to be big nightmares.

For the first batch that we shipped we did things in a very ad-hoc way. I used Microsoft Word’s Mail Merge feature to print many of the labels; for international orders I wrote a little Access application that filled out forms on the USPS web site, and so on.

The actual shipping turned out to be quite difficult. It was amazing how small ergonomic problems turn into huge headaches when you’re shipping thousands of orders to 60 different countries. For example, the customs form generated by the USPS web site needed to be signed. We had tons of these. We tried to “sign” them by running them through the laser printer again and printing a couple of signature images in the correct place. This worked OK on one printer, but every other printer in the office jammed up when you gave it paper that had already come out of a laser printer. Why is that? Eventually, of course, the printer which could print signatures ran out of toner. It was a Dell. Dell doesn’t let office products stores sell their toner. The only way to get toner was to order it from Dell and pay a lot of money for overnight delivery.

My estimate was that it took about three minutes of labor per order, and we only got it that low because we were batching them up. Worse, it took someone skilled in SQL queries and Mail Merge to create the labels.

Last weekend, I thought, there’s got to be an easier way. I spent some time researching tools on the Internet for making this stuff easier, and on Monday, I started building our new Super Duper Shipping System, aka FogShip, which took about three days to set up, program, and debug.

Here’s what the physical system looks like.

The whole process is controlled by a single computer running Microsoft Access. The shipping data itself lives in SQL Server.

Every day when you’re ready to ship things, you click on a button in Access which causes a stack of packing slips to print out. A typical packing slip is shown at right.

Why Microsoft Access? I was skeptical myself. I’ve never, ever used the Access “report” feature in my life; I always imagined it was intended for old-fashioned people who need to have their data printed out for some obscure reason and can’t fathom the idea of looking things up on a computer.

Well, it turns out that Access reports are perfect for generating packing slips. This is what they were born to do. Since the data is already in SQL Server, it took no more than a couple of hours to layout and design the report exactly the way I wanted it.

Two other things you’ll notice about our packing slips, learned from hard experience. First of all, instead of printing them on a standard-sized sheet of paper, I printed them on 5×8 cards. This has two big advantages. First, they fit into the envelopes we use for shipping most books, CD-ROMs, and DVDs without folding. This really makes a difference when you have a lot of shipping to do. Second, because the card stock is a little bit stiffer than paper, it’s easier to shove them into the envelope.

You’ll also notice the bar code printed in the top corner. There are many different schemes for printing barcodes, but if you’re printing your own barcodes, the easiest barcode system to use is something called Code 39. Other barcode systems want you to add a checksum at the end of the barcode and that means you have to write a little bit of code to calculate the checksum. But Code 39 embeds the the error correction protocol in each letter, which means that you can get a Code 39 font that works just like a font without writing any software.

When you’re ready to start shipping you grab a packing slip and scan it.

I didn’t really know which barcode scanner to buy, and no matter how hard I tried, I couldn’t find any online reviews of barcode scanners. I did find out that the kind of scanner I wanted was a USB keyboard wedge scanner. That means that it has a USB port, which makes connection simple, and it behaves exactly like a keyboard as far as your computer is concerned. I also found out that I wanted a laser scanner, not a CCD scanner, because it works at a distance. I ended up buying a Wasp WLS-9000 along with the stand which you can see in the picture. So far it’s been great.

Setting up the barcode reader was the easiest thing in the world. Plug it in to a USB port, and you’re done. It comes with a whole book of barcodes that you can scan to configure it to behave in many interesting ways, which you can throw away. Without any setup, when you scan a barcode, your scanner will type the letters that barcode represents on the keyboard and press Enter.

In FogShip, scanning the barcode launches a VBA process which does three things.

First, it marks the order as shipped in our online database, so the customer gets an email to let them know their order is on the way. Technically speaking, it would be more accurate to tell them their order shipped when the post office actually came by and picked up their order, but that enhancement will take more programming and will have to wait.

The second thing that happens when you scan the barcode is that it displays the information from the packing slip on a second large screen in a giant font. That way you can immediately shove the packing slip into the envelope… you don’t have to keep it around any more just to keep track of what you’re packing. This was a lesson learned from the first round of shipping, where we had to keep the packing slip outside the envelope until the last minute so that we would know what to pack and what the shipping method was; keeping the packing slip out until the last minute was just one more annoying thing that made the whole process less fluid.

Third, it prints a mailing label, and, for international orders, a customs slip.

To print USPS mailing labels with postage, you really only have three choices. You can use the USPS website (“Click ‘n’ Ship”), but that is designed for home users shipping one or two items. It is not designed to be automated and it always prints lots of instructions which you throw away. Since it relies on Adobe Acrobat Reader for the label, the process of printing is cumbersome and doesn’t work well unless you use expensive, full page 8.5 x 11 labels from a laser printer. Anyway, Click ‘n’ Ship is not the way to go.

The other two choices are Stamps.com and Endicia. Both of them have ways to programmatically print a label.

Endicia had a couple of advantages over Stamps.com for our application: it had a decent tool for laying out custom forms and labels exactly the way you wanted, and it had all the documentation on their website. Stamps.com required you to sign up for a developer program, so it took a while to get the documentation we needed, and we needed that thing yesterday, so we just went with Endicia.

Endicia’s system for printing labels couldn’t be easier. You create an XML file with all the shipping information and launch their app with the name of that XML file on the command line. It spits out a label and exits, creating a new XML file with tracking information and status information. The stamps.com system, as far as I know, is based on ActiveX, which would have worked fine for our application, too.

Now… let’s talk about spitting out a label. If you’re doing more than three or four labels a day, you’ll want a label printer, and if you want the whole process to go quickly, you’ll want a really fast label printer, not a laser printer that has to warm up. Everybody recommended Zebra which makes high-end, industrial strength label printers. We went with the Z4Mplus, which spits out a 4×6 label in less than a second. Not only that, it peels the label for you if you get the optional peeler (sign me UP!) This thing is really, really cool. And if you get Direct Thermal labels, you don’t need to buy a ribbon or ink or toner or anything… it sort of burns the print onto the label.

Next problem: customs forms for international orders. A customs form looks like this:

The green bit on the left is a sticky label which you peel off and stick to the envelope. The white part is just plain paper, attached via perforation. You have to fill out the whole thing with details like price, an exact description of the products, how much they weigh, and you have to sign both sides. Filling these things out is a pain in the butt.

The first trick was to find a printer that could print on 4″ x 7″ paper, the size of the customs forms, without messing up the sticky label part. It also had to be able to print pretty close to the edge of the page, it couldn’t depend on manually feeding each form, and it had to look good enough so that an image of the signature wouldn’t look too fake. It had to be fast at printing and not take a long time warming up, or it would slow down the process.

The printer we found which met all these criteria was the HP LaserJet 2420. We bought two of them (the other one prints the packing cards). It can hold about 100 blank forms in the envelope tray… it turned out that these forms are almost exactly the size of Monarch envelopes which is a standard format for most programs.

The setup software that comes with this printer is completely awful and crashed trying to install on a brand new Windows XP computer. It took several hours of fooling around with the drivers before this thing would print properly. In the meantime, it was printing, but super-slowly, and we couldn’t figure out why. I’m pretty shocked that in this day and age HP is still shipping printers that just can’t be set up by normal humans.

Back to Microsoft Access, where I created a report for the customs form. Using a ruler I carefully measured the position of each of the fields on the paper form and set up the Access report accordingly. It worked great, even producing a decent facsimile of a signature on the signature line!

The last step in the process is attaching any other flotsam, like Airmail or Priority Mail stickers.

And, tada! You’re done.

Shipping an international order now takes about 35 seconds, down from 3 minutes, and can be done by anyone, whether or not they have SQL and Mail Merge skills. Domestic orders are even faster since they don’t need customs forms. Most of all, it’s all really fun.

Project Aardvark, if you’ve been following along with the blog, is our summer interns’ new product. We’ve got four interns here (three in development, one in marketing) putting together a complete product from beginning to end. Now that they’ve officially announced what it’s all about and we’re about to start the first beta, I can bring you up to date on the project, which is more or less halfway done.

The Idea

If you’ve ever tried to help your technologically-challenged uncle fix his computer problems over the phone, you know what a pain in the butt it can be to try to walk him through the fix.

“Click START”

“What?”

“Start. Click Start. It’s in the bottom left.”

“I have C – T – R – L in the bottom left.”

“The bottom left of your screen.”

“Oh. OK, I clicked it.”

“OK, now click RUN.”

“What?”

“On the menu that came up. Click RUN.”

“It’s not there.”

“What do you mean it’s not there?”

“It’s not there. I don’t have a RUN.”

“What do you see? Read me everything you see”

“Recycle Bin… My Computer… Anna Navratilova J P G…”

“No, on the menu.”

“What menu?“

“The menu that came up when you clicked start.”

“When I what?”

This is when you give up and realize that something that could take you 10 seconds to fix in person is about to become a two hour nightmare during which you’ll alienate your family, lose sleep, tie up the phone line while your Auntie Marge is stuck on the turnpike with no gas and can’t get through to your uncle to come rescue her, and curse your lot in life. Just because you’re a programmer doesn’t mean you have to be the help desk for a dozen friends, relatives, and the people in the apartment next door. Does it?

That’s the general idea behind the new Fog Creek Copilot^SM service. In a nutshell, you go to copilot.com and get an invitation code. You tell your uncle to go to copilot.com and type in that same invitation code. You each get a little program to download and run. When you run the program, your uncle’s computer screen shows up in a window. When you move your mouse, his mouse moves. When you type something, it appears on his computer. Etc. And now you fix the problem and log off, and peace is restored and your aunt gets home safely and your uncle dances at your wedding instead of boycotting it and holding up unpleasant signs across from the hotel where said wedding is taking place.

But but but…

Yes, similar services already exist. That never stopped me before. I’d like to point out that Fog Creek has been doubling in revenues every year mostly thanks to bug tracking software, and it’s not like we invented bug tracking software. There are a few things our product will do better than the competition, but mostly we just want the Fog Creek Copilot experience to be shockingly seamless. It’ll be totally secure, it’ll be cheap, it’ll be painless, it will work through firewalls on either side so you can help mom at home on her firewalled DSL from behind your NAT at work without a hiccup. We even made it so that the little software program you download is totally self-contained, totally pre-configured, and deletes itself when you’re done so you can feel more secure about the whole episode. There’s no commitment; you don’t have to sign up or create an account and remember a password; you can even make your uncle pay since, after all, he’s getting the benefit.

For the geeks in the audience, the service uses a highly customized and optimized version of VNC, but it also requires a customized “reflector” service that we’re building which sits outside of any firewalls. The idea is, since you can’t connect into mom’s computer which is behind a firewall, she’ll connect out to our server, you’ll connect out to our server too, and the reflector will forward data back and forth between the two of you.

How’d you get the idea?

For the last few years we’ve been supporting FogBugz customers using a similar scheme, although it’s a bit of a pain to set up. Our customers have to follow 7 steps to allow us to control their computers, and we’ve found that walking people through these steps on the phone takes an average of 5 minutes. With the Fog Creek Copilot service we’ll just tell them to check their email and click on a link and hey presto! we’re fixing their computer.

So the original idea was to use this for tech support. But when I told the idea to the interns, two out of four said, “yeah, I could use something like that to help my mom.” That’s when we realized there’s a huge world out there of Informal Tech Support… lots of people trying to help Uncle Leo who can’t use products like VNC because of the firewall problem. So we changed the focus of release 1 to be the casual and home user instead of tech support departments.

The Name

We had a list of important criteria for the name, but the most important one was that when someone read the name to someone else over the phone, it would be extremely likely that they would get it right. This ruled out names that are weirdly spelled, names which could be easily confused over the phone (for example “m” and “n” are almost impossible to distinguish), and names that have different possible spellings. We went rather too far along the process of investigating the name “Fixant” (complete with a very cool drawing of an ant holding an ethernet cable) before I just got everyone together in a room for a half hour of brainstorming, when we finally hit upon the idea of “Copilot.” I can’t even remember who thought of it. The idea of brainstorming is just that you shout out ideas, which stimulate other people to have ideas, and you put them all up on a whiteboard.

Well, there are a couple of dozen products named Copilot, many with registered trademarks, so our trademark lawyer advised us to use Fog Creek Copilot which would eliminate any possibility of confusion with those other Copilot brand products. The point of trademark laws is that what you’re not allowed to do is create any confusion or potential confusion as to the origin of your product, and sticking “Fog Creek” in front guarantees that, but we have to be religious about always using the full name. I didn’t really mind, having started my career working on products like Microsoft Excel, Microsoft Visual Basic for Applications for Microsoft Excel, etc. etc. After a few weeks on the Microsoft Excel team if you ever saw the word “Excel” without a “Microsoft” in front of it, it looked nekkid.

We bought the domain name for more money than we spent developing the first version of FogBugz, oy gevalt, but it is a really good name — easy to spell, pronounce, and it even sort of suggests what the product does, which makes it more memorable.

The Conference

For some reason, a long long time ago, I had agreed to give the keynote speech for CFUNITED, a conference about ColdFusion.

“I never used ColdFusion!” I protested.

“Don’t worry. Nobody has. The biggest sponsor of this conference is Microsoft, who have a huge presence trying to get the ColdFusion developers to switch to VB.NET,” the organizers told me.

As luck would have it, the conference timing was perfect for the first feature-complete version of the interns’ code. It gave the team a deadline to work to. At the conference itself, we set up a Fog Creek booth and the interns gave demos to several hundred attendees who wandered by.

This was the first trade show Fog Creek had ever attended. The truth is, a trade show is not a very cost-effective way to reach potential customers. Given the cost of travel, hotels, the booth, a thousand bucks for nice brochures, and everybody taking a week off of work, it’s a really expensive way to get in front of prospects, especially since I can write an article on my website and get in front of 1000 times as many people.

But that’s not really the point: the point is to have interactive experiences with your customers. You can try out lots of different pitches and really listen to how people respond to them, which is something you can’t do in non-interactive marketing like web sites and magazine ads. I learned this from Eric Sink, who wrote a great article on the topic, Going to a Trade Show.

We went down to Washington in two big SUVs with all four interns, one of the FogBugz developers, Brett, who gave demos of FogBugz, and me. Our booth probably looked just a little bit too much like a science-fair exhibit, but, whatever, it was our first attempt. Next time we’ll know to make backdrop posters that stretch right out to the edge of the backdrop which looks a bit more professional, and I’ll remember to bring a lucite brochure dispenser instead of arranging the trifold brochures artfully in the shape of an Aardvark.

But that’s not really a big deal. What was a big deal is that we got to talk to hundreds of potential customers, and, wow! the response was just incredible. At the very best, the response we got was, “I need a thousand of these yesterday for my whole team.” Almost everyone was impressed by the product and knew that they wanted to use it. A very small number of people were aware of other competitors and other solutions to the problem, but mostly people gave us very positive feedback. More importantly, after spending two days pitching the product again and again to lots of different people we learned the most effective ways to present it. We learned that the best way to present it was not to start with, “You know VNC?” This drew glassy stares. The best way was to start with a typical remote support scenario. “Your mom calls you up. She says her screen is half grey. You have no idea what the heck she’s talking about.”

The Beta

We got back last Friday and immediately started working on the beta. The goal of this summer internship is to be shipping to paying customers by the end of the summer, and I didn’t want our interns to go home after “mostly” finishing the code, leaving us permanent employees to debug until next February, so we’re working on a really compressed schedule. Over the next few weeks, we’ve got to:

• Deploy the site and service on our web farm
• Launch a controlled, private beta so a few people can start trying the service and we can figure out if it works in the real world. Yaron is taking beta applications now.
• Launch a wider public beta
• Do a round of usability tests in the lab. The nice folks at TechSmith have a product for usability testing called Morae, and they’ll be coming out to New York and helping us organize and run the usability tests. Morae lets you set up a complete usability lab using nothing more than their software and a little webcam
• Start serious QA.

So far we’ve really been hitting the schedule so I’m pretty confident. In the meantime, you can:

• Follow along with the Project Aardvark blog!
• Apply for the beta! If you’re accepted, you can try the Fog Creek Copilot service yourself!
• If you’re in New York, come to the Project Aardvark Open House! You’ll get free wine and cheese and you can pepper the interns with questions about sockets programming in person. The open house is Thursday, July 14, 2005, from 5:30 PM – 7:00 PM, at Fog Creek Software, 535 8th Ave., 18th Floor, New York.

I was really hoping to ship FogBugz for Windows and FogBugz for Unix on the same day, but it just wasn’t gonna happen. There were a bunch of little details we needed to get right on the Unix side and with over a year of development we had accumulated a few little things that Thistle didn’t compile perfectly. And it seemed a shame to hold up the Windows version (90% of the market) for the Unix version (10%). So on February 23, 2005, at 7:35 PM Eastern Time, FogBugz 4.0 for Windows went live.

I thought to myself, ah, now I can relax.

I have no idea why I thought that.

I should know by now that shipping software is the beginning, not the end.

We promptly got whacked. The phone lines lit up like a New Mexico license plate frame. Our weekly sales tripled. We could barely keep up with the phone calls and email inquiries. This is a good problem to have, and I don’t know why I did not predict it, but even if I had, I probably wouldn’t have done anything differently: I believed what Eric Sink wrote in The Hazards of Hiring: “Don’t fill a position until after the need for that position is painfully clear.”

And we won the Jolt Award, which is unbelievably cool (4.0 was too late so the award was for version 3.1):

In the meantime, we’ve got another half dozen things on our plates at Fog Creek Software. We’re starting to get the kind of gigantic customers who want things just a little bit different, and they’re willing to pay so much money for it you’d be insane not to give in.

We’re working on rolling out a new scalable stack o’ servers to handle the demands for the next five years.

We’re gearing up for the coolest summer internship program you’ve ever heard of… our four interns will develop a brand new product from soup to nuts, and get the darn thing up and running and earning real money from real customers by the end of the summer — something that’s challenging enough without a filmmaker running around making a documentary about it, which, I hope, does not turn out like The Restaurant, Blow Out, or, god forbid, The Office. My David Brent streak is a mile long and will need to be actively surpressed. “I see myself as a friend first, probably entertainer second, and boss third…”

And we really, really need to hire more people, no matter what Eric says, which means we’ll need a bigger office, so that’s more work too, and life goes on.

A ridiculously small portion of the energy it takes to make a commercial software product actually goes into the writing of lines of code. I would estimate that out of every 100 calories expended by the Fog Creek team:

25 calories are spent on customer service
55 calories are spent on debugging, beta testing, and minor tweaks
8 calories are spent on marketing, including the Fog Creek website
5 calories are spent reading college kids’ resumes and interviewing said college kids
5 calories are spent on code that never ships, such as the online demo and the online store

Leaving just:

2 calories spent on actually writing new lines of code that ship to a customer.

This, I think, helps explain why so many software companies started by programmers fail: programmers are really good at writing news line of code, they might be good at debugging, but nobody ever taught them how to do marketing or customer service, and they are probably horrible at graphic design.

Before a consumer will buy your software product, they’ll evaluate all kinds of things to decide if it’s a real product that will meet their needs:

• They’ll evaluate the quality of your website.
• They’ll look for discussion groups to see if people are actually using the product, and if the people who have problems are getting prompt resolutions.
• They’ll look to see if there is a third-party support infrastructure, like books.
• They’ll evaluate your reputation.
• They’ll search the web to see if you have positive buzz.
• They’ll see how long you’ve been around, and they’ll try to evaluate whether you’re profitable and successful and likely to stay around to continue supporting the product.
• Oh, and if they have a few minutes left over, they might actually look at your product itself to see if it works.

And that explains why people buy overpriced but useless shelfware that costs $millions and doesn’t really work. They rationalize that if they install it, they’ll be shielded from criticism and other mercurial notes of anger.

Anyway, before we could launch FogBugz, even after the final shipping bits were perfect and ready to go, we had a list of things we wanted to have ready to make the product experience complete. For FogBugz 4.0 the big things were:

1. Professional-quality graphic design in the user interface
2. An online demo that included every feature of the product
3. An online movie that introduced the product for couch potatoes
4. A great marketing website
5. Getting at least one book about FogBugz into bookstores
6. Making a real physical product available on CD-ROM

Graphic Design

Human emotions can be really, really superficial. In particular people ridiculously overvalue aesthetics and beauty when evaluating products. It’s one of the reasons iPods, and, for that matter, Keanu Reeves, are so successful. If you have a product that is functional but not beautiful you have a really, really, really, really, really, really, really, really, really, really big handicap to overcome against the product that is beautiful but maybe not so functional.

Given the importance of this I spent days looking at portfolios of web designers. I happened upon CSS Zen Garden, which was new at the time, the effort of Vancouver-based graphic designer Dave Shea who wanted to prove that CSS did not imply ugly boxy design. In those days there were only a few designs available on Zen Garden and a lot of them were done by Dave himself, but they were all stunning. (Here’s one I particularly liked).

So we hired Dave to do a redesign of FogBugz.

Before:

After:

OK, now all the programmers in the audience are saying, “what? I don’t get it?!” I know. I don’t either. But the new user interface looks and feels, well, 5% more “quality,” whatever the heck that means, and it’s all very nebulous and vague but I can tell ya everyone has been saying it’s the cat’s pajamas, so I’m happy.

Online Demo

The online demo speaks for itself, but we had to do a lot of work to make it possible to host hundreds or thousands of instances of FogBugz in one directory.

In particular we were afraid that one spark of good publicity would result in hundreds or thousands of people setting up new trial accounts simultaneously, which would whack the poor servers and result in a lot of potential customers thinking FogBugz was slow simply because they were trying it on an overloaded server. So we changed the trial signup procedure to include a throttle that, under peak load, allows us to slow down the creation of new trials. Occasionally, it may mean that a potential customer has to wait a few hours to try out the software, but it’s better than everybody having a bad experience.

The Online Movie

One product I just can’t stop singing the praises of is Camtasia Studio, by the folks at TechSmith in Okemos, Michigan. It lets you “film” your computer screen, all in software of course, then edit the film, add narration, then squeeze it down to a very compact flash movie you can put on the web. This is a beautiful piece of software. It does exactly what you need, works the first time, and comes with great documentation you’ll never need.

I used Camtasia Studio to record an introduction to FogBugz 4.0 in movie form. The whole project took me about one day. If I had to make another movie, it would only take a couple of hours, as I learned many tricks along the way. For example the first recordings I made sounded terrible. Our offices are in New York City and a lot of street noise drifts up to us here on the 18th floor. It’s not enough to notice during the day but when you’re trying to record something, the muted sirens in the background are shockingly unprofessional! To make things worse, we have hardwood floors throughout the office and don’t have those yucky acoustic tiles on the ceiling, so the office is acoustically live in a way which made the movie sound like I was in a big abandoned warehouse, down by the river, waiting for Kojak to arrive with a lollypop.

I gathered up all the rugs and pillows I could find at Fog Creek and moved them all into my office. With a big rug on the floor and giant sofa pillows lining every wall, the sound quality was vastly improved. In fact my office was so pleasant with all the sound-absorbing stuff that I realized it’s not enough to have quiet offices; you want to have acoustically dead offices as well. For our next office space I think I’ll specify thick carpeting for the private offices, even if it’s not as cool as wood floors. We’re seriously considering hiring Aviva Stanoff to come up with some kind of framed pillow arty thing to put on the walls to deaden the sound even more.

The Website

Um, yeah. We threw this together at the last minute. We did manage to find a few good-looking FogBugz customers to give us testimonials so we could replace the crappy stock photography with pictures of actual humans, and Dmitri found a great old picture of some craftsmen making violins. I spent a lot of time putting together a screenshot tour because a lot of people look for the screenshots first when they’re learning about a new product.

FogBugz, The Book

Different people learn different ways. I always want to dive in and start using a product, but some people prefer learning about a product in a classroom, and others prefer to read a book. I’ve even heard people say they won’t use any development tool until there’s a genuine book about it in bookstores. A real book from a real publishing house is one of those things that makes a product “feel” real.

I had been incredibly impressed by Mike Gunderloy’s classic book Coder to Developer, about all the things that a full-fledged software developer needs to do above and beyond the basic work of writing lines of code, and Mike has been working with FogBugz since version 1.0, so I persuaded him to write a book, which Apress published.

The reason I like Mike’s book so much is that it’s not just a rehash of the online documentation, which is pretty detailed to begin with, and it’s not just a big wasted list of steps for filling out dialog boxes. Instead Mike’s book focuses on how to manage software projects using FogBugz effectively, so it adds a lot of useful material. In fact I liked the book so much I ordered a thousand copies, which we’re selling along with FogBugz. Probably 20% of new FogBugz customers buy a copy or two of the book, and it’s also good to have around as a handout for potential large customers.

(By the way, if you’re a Mike Gunderloy fan like me, you’ll be delighted to see that he has recently relaunched his website Larkware as a daily source of news about programming technologies.)

FogBugz, The CD-ROM

One of the nicest things about selling software is that there is no physical product, but I’m kind of old fashioned, and whenever I buy software online, if there’s an option to receive a physical disk, I usually take it, just so I can have something physical up on the shelf that I can find if I ever need to reinstall the software. I can’t tell why but somehow having the option of a physical product makes the software feel more tangible and therefore worth more money. It’s a great mystery to me why this is the case but I’m sure anthopologists would not be surprised.

Since we already decided to sell physical products, i.e., the book, we needed to get set up for shipping physical products anyway, which meant inventory, postage meters, UPS accounts, and all that stuff, so I figured, against my better judgement, that it couldn’t hurt to make CD-ROMs available as well.

The FogBugz CD-ROM is entirely manufactured at Fog Creek, using a robotic combination CD-ROM recorder/inkjet printer which records and prints the CD-ROM without human intervention. We put them in DVD boxes, which are a little bit classier than CD-ROM Jewel Boxes. So far it looks like we’re shipping two or three a week to customers who opt to pay an extra ten bucks for physical media. It’s hard to tell if it’s worth it. Somehow I think that having a real packaged product makes FogBugz “seem” more real even if almost nobody orders it, but again, maybe that’s just me being old fashioned.

Tomorrow, the aftermath!

One day in the summer of 2003, after my friend Uday had finally told me for about the 17th time that he would have bought FogBugz but they didn’t have any Windows servers to run it on, I decided to figure out a plan for getting a Unix version. We had looked into Chilisoft, an implementation of ASP and VBScript that runs on Unix, but we didn’t like that idea much because at the time it cost something like $1000 per server and that wouldn’t be acceptable to our customers, who could have bought a whole ‘nother Windows server for not much more than that.

One morning walking to work I thought, gosh, it would be cool to have a summer intern, and I could tell the intern to write an ASP compiler that generated PHP as its output. Then we could produce a Unix version of FogBugz from our source code automatically, and as we added new features to the ASP version, they would magically show up in the PHP version without any more work.

Lo and behold that day this kid named Jimmy calls up the office. He’s just back from Junior year in Italy and he needed a summer job. When I gave him one of my (unpublished!) interview questions (don’t look for it online, you won’t find it) he answered it in exactly the same way as I had answered the same question when Brian MacDonald asked it to me for an internship at Microsoft back in 1990. All the same false starts, the same initially-wrong answer, and eventually, the ideal answer, Jimmy was sounding like a clone of me.

Against my better judgment, knowing myself, we hired him for the summer, and thus was born the Fog Creek Summer Internship program.

Jimmy didn’t know PHP, or ASP, or Java, really, but we thought Java was a good implementation language for an ASP to PHP compiler, so he dealt. Within a few weeks we had something up and running. By the end of the summer it was pretty decent, running FogBugz like a champ, although after Jimmy went back to school, getting all the tiny details right took Michael another couple of months.

I keep calling this thing an ASP to PHP compiler, which leads many people to email me saying “don’t you mean translator?“

Let me explain.

In computer science jargon a translator IS a compiler. It’s exactly the same thing. Those are synonyms.

In common usage, people think of a compiler as generating machine code, but it doesn’t have to. Java and .NET compilers generate bytecode, for example. And Stroustrup‘s first C++ compiler, called Cfront, actually generated C code because C was portable and this made it possible to run C++ on any system that had a C compiler. When you’re inventing a new language it’s a waste of time generating machine language output for every architecture when you can just generate nice messy C code and let cc generate the machine language and do all the back end and optimization stuff. Cfront was terribly confusing to a lot of people because it was often referred to as a “preprocessor,” so people got the feeling that C++ was just a clever set of macros that use the old-school C preprocessor cpp to generate C code. Nothing could have been farther from the truth. There’s no way you could use the simplistic C preprocessor to convert C++ to C. Cfront was a complete, genuine compiler, with all the classic parts of a compiler: a lexer, a parser, an abstract syntax tree, etc., but instead of outputting machine language which would only work on one CPU, it output terrifying, yet perfectly portable, C code.

And that’s what our ASP to PHP compiler, Thistle, does. It’s a complete compiler, not a bunch of glorified regexps, which lexes and parses a VBScript/ASP program, creates an abstract syntax tree in memory, and then generates an “executable program” which happens to be in PHP. So I’ve been calling Thistle a “compiler” sort of to make a point, but mostly because I’m a pedantic windbag who can’t resist the opportunity to teach a little lesson to the younguns who think that a compiler has to generate machine code.

Windbagginess aside, I always get a few questions about Thistle, so I’ll answer the common ones here:

Q. Why didn’t you use asp2php?

The quality of the “translation” produced by asp2php is nowhere near good enough to produce production quality code. On samples we fed it, almost every generated line contained an error. There are some really subtle tricks to VBScript and PHP which asp2php seems to completely punt on.

(OK, you really have to know? In VBScript you can assign an array to a variable one minute, and one minute later assign an object with a default method taking one integer argument to the same variable, so the expression a(1) would translate as $a[1] in the first case and $a->$default(1) in the second case, and it’s the same expression, and this fundamental problem was completely punted on by the designers of asp2php, probably justifiably so. In a moment I’ll tell you how we solved it.)

Furthermore, we had a hard-and-fast requirement that the generated PHP code had to work without further tweaks. That’s because the ASP code is a living thing which changes all the time, and we constantly regenerate the PHP code automatically. If there were any manual tweaks required to get the output of Thistle to work it would dramatically increase the headache of getting a PHP build. So we implemented all kinds of conditional compilation tricks, and since we controlled the compiler, we could build in all kinds of intelligence to make sure that the generated code was exactly what we wanted.

Q. Why don’t you commercialize this great Thistle thing if it’s so great?

Thistle is a compiler that only has to compile one program: FogBugz. That means it doesn’t need to include any logic for any features that FogBugz doesn’t use. There are lots of library functions we don’t have to translate because we never call them. And there are conventions in our code that we always adhere to that allow the compiler to take big shortcuts. That means Thistle would not work so well for anyone else off-the-shelf.

Remember the problem of how to translate a(1), which could mean “look up the 2nd element of array a” or “call the default method of the object a passing the argument 1″ depending on what type a contains at runtime? This really matters, because we use arrays, and because we use the built-in class RecordSet all over the place, doing things like rs(1) which is short for rs.Item(1).Value, and since VBScript is latebound there is no way to know what code to generate in PHP until runtime, and that’s too late! The only correct thing to do in PHP would be to generate code that checks the type of a, and decides, at runtime, whether to do an array lookup or a method call. This is messy and slow and would suck big rocks in the kinds of tight loops where you tend to be using arrays.

How did we fix it? Well, thanks to Hungarian notation, so callously dissed by developers who do would not recognize a superb coding convention if it walked up to them on the Shanghai Maglev train and shook their pants leg, every recordset at Fog Creek starts with the letters “rs”. And Thistle looks for the rs and say, “ah, this is a recordset, you’re not looking for an array value, you’re calling the default method,” and generates fast code. Based on your age you will either call this an evil hack (if you’re young) or an elegant hack (if you’re old); in either case it’s a huge optimization made possible by the fact that Thistle only has one program to compile. Outside of Fog Creek it wouldn’t work. All hail Hungarian notation!

One more reason we don’t commercialize it: it’s a competitive advantage to us allowing us to create server-side applications that run on Windows OR Unix (or Mac or Solaris or Linux) with a single source tree. But that’s secondary.

Q. Are you serious about Hungarian notation?

Yes, it’s standard at Fog Creek. We try to use something called Apps Hungarian notation, as invented by Simonyi, not the grotesque bastard Systems Hungarian notation, misinterpreted by Petzold and the entire Windows team. You can still read the original paper.

In Apps Hungarian you still have the prefixes, but they are supposed to add semantic information, not merely repeat the type of a variable. So for example when you have a buffer size, the variable should be named “cb”, which means “count of bytes”. Whenever you’re writing C++ code that needs to deal with all kinds of strings, it’s really nice to be able to look at the variable name, and if it’s a psz, you know, oh, look, it’s a pointer to a null-terminated string, but no memory is allocated! Or if it starts with rgch, you know it’s an array of characters, with memory allocated, or rgwch, you know it’s an array of Unicode (wide) chars, or if it starts with ix you know it’s an index to something (or a primary key in SQL), etc. etc. It’s really nice when you’re manipulating unicode that cch means “count of chars” while cb means “count of bytes” since a char is not a byte, and if you ever got confused, the variable name sorts you out. And it’s really nice when you don’t remember if you have a pointer to a string or a pointer to a pointer to a string and the “p” or “pp” at the beginning sorts you out, too. Anyway, Hungarian is widely and correctly reviled by people who don’t see the point of putting dw in front of all your longs, although even that was useful in its day, but using “c” for a count or “ix” for an index is quite helpful.

In SQL tables we use very light Hungarian to indicate the type of fields: s for strings, dt for date/time, c for a count, ix for a key, n for a number that is not a count of something or a key of some sort. The most useful aspect of this is that if a table is named Bug then you can be certain its primary key is ixBug, so whenever you’re looking at SQL anywhere in the database if you see ixBug you know it’s a foreign key pointing to the Bug table without bothering to look it up. And now without even thinking if you see this snippet of SQL: “join Bar on Foo.ixBar = Bar.ixBar” you know exactly what’s going on, and if you see “join Bar on Foo.ixFred = Bar.ixBar” then you know it’s probably wrong code because with a little bit of practice the mismatched ixFred/ixBar just jumps out at you.

I’m especially driven crazy by people who keep blabbing on their blogs that Hungarian makes code “less” readable. Yes, it’s less readable if you don’t know the convention, but it takes about 10 minutes to learn the convention and all the important prefixes, and people who whine about Hungarian making code “less” readable have obviously never done this. If you know what the prefixes mean Hungarian makes code more readable because every variable gives you a little bit of free, extra useful information you don’t have to go look up.

Q. How did you handle all the runtime libraries ASP provides?

We reimplemented them — either in VBScript, which gets translated to PHP, or directly in PHP. In most cases there is a very close PHP equivalent so the reimplementation is one line.

Q. What about COM objects?

That was the most work. For our own COM objects we wrote portable C++ code and we compiled that to .so’s for Unix. For one of the major third-party COM objects that we use, we reimplemented it completely in PHP using a built-in PHP library. For the built-ins like Server, Response, Request, Recordset, etc. we created our own classes which usually just call underlying PHP functionality. And our implementation of Server.CreateObject knows all the tricks and just implements a giant select statement that constructs the right kind of object so we don’t need an equivalent of COM class factories.

Q. Was it worth it?

Probably. FogBugz for Unix is about 10% of our sales. The effort to create it was probably 20% above the effort to create FogBugz for Windows alone. A lot of our customers are happier that we run on Unix even if they’re using the Windows version, because they know they could switch servers at any time. And finally we have a certain amount of confidence that if Microsoft ever decided to stop making VBScript/ASP work on their server OS, it would be a very simple matter to change Thistle to generate the new flavor of the week … be it Ruby.Net, Eiffel, C#, or this Language.

Tune in again tomorrow, when I’ll talk about all the other stuff we did for FogBugz 4.0 to make it a complete product, not just a computer program.

A long time ago I wrote an article called What is the Work of Dogs in this Country? about the benefits of eating your own dogfood, a quaint expression in the software industry that means “using your own product.”

We’re totally devoted to the idea of using our own software as much as possible… all my writing is done inside CityDesk (a very unstable developer build, at that, which drives everyone completely crazy), all our development is planned and tracked in FogBugz, of course, and even our customer service operation is entirely supported and tracked with FogBugz, which is why when you email us you always get a reply.

Well, almost always.

About a year ago, I noticed that people were emailing us, sometimes, and not getting responses.

This was not good.

I dug up their messages in FogBugz.

RESOLVED: SPAM, it said. Someone handling the customer service queue had decided that the email was spam and closed it without responding.

When I looked closer, it was usually because the subject line sounded kinda spammy or was blank, but sometimes it was just the proverbial Operator Error.

In the meantime I had been using SpamBayes for my own personal email. This is probably the best implementation of what is probably the best spam filtering algorithm out there: Bayesian filtering, invented by Paul Graham and first published in August 2002 in his seminal article A Plan for Spam. Bayesian filtering learns from experience. When it makes a mistake trying to classify an email as spam or non-spam, you correct it, and it looks for clues in that message so it can do a better job at classification in the future.

And I thought, gosh, here I am paying a college graduate to go through email and decide whether or not it’s spam, and he’s flagging 1% of email incorrectly, when there’s this algorithm called Bayesian filtering which in my experience has far fewer false positives (something like 0.01% false positives). So while everybody is worried about how spam filters might inadvertently delete the proverbial crucial email from a customer, in practice, in the presence of lots of spam, human beings are far more likely to delete a real email than a well-implemented Bayesian filter.

The other thing about the Bayesian algorithm, by the way, is that it has to be implemented in the email client, not the server, because it needs to be taught about what one particular person’s legitimate email looks like. For example when we get email at Fog Creek, the word “FogBugz” in the body of the email is strong evidence that the email is not spam while the word “mortgage” in the body of the email is strong evidence that the email is spam. However if you’re a real estate agent or bank, the word mortgage probably occurs all the time in legitimate email. So basically Bayesian filtering is not going to work unless it knows what messages you deleted as spam and which suspicious messages you recovered. The Ipswitch guys who make a mail server tried to implement Bayesian filtering on the server, because that’s what they have, and frankly that system just doesn’t work.

Since spam filtering needs to be done on the client, and FogBugz is an email client, we decided that spam filtering needs to be done in FogBugz.

Now, all that probability stuff is too hard for me, so we asked Summer Intern Ben Kamens to figure it out. In a matter of weeks he had a beautiful, speedy implementation of Bayesian filtering up and running in FogBugz.

Bayesian filtering is really just a specialization of a long established AI algorithm for sorting documents based on training, and we thought, gosh, why not generalize the algorithm so that in addition to sorting incoming email into “spam,” “suspect,” and “ham,” it also sorted all the ham into piles, for example, “sales,” “tech support,” and “job applications”?

Ben spent a few more weeks studying this and came up with the idea of using a tournament algorithm. He’s a college athlete. They think of the world in terms of tournaments. It worked quite well. With 24 hours of training I had it sorting my own inbox into personal email, Fog Creek email, Joel on Software email, Joel on Software translator coordination email, Subversion checkin comments, bug notifications, and sales reports. Frankly it did an astonishingly good job with 100% accuracy on the obvious emails, like the checkin comments that all had the same structure, and well over 90% accuracy on the hard ones, like telling personal email from Joel on Software reader mail.

I was tempted to try and get it to sort email into messages from Republicans vs. messages from Democrats, but that was pushing my luck.

By the way, the paper Ben wrote at the end of the summer describing his algorithm is here (PDF). This paper is the ultimate rebuttal to those grumpy people who email me, barely able to conceal their disgust, saying, “why do you need to hire such smart people to work on bug tracking software?”

The cool thing about the implementation in FogBugz is that if the sorter makes a mistake, it’s OK. You just correct it and get on with your life, and it learns from its mistake. But in the meantime it saves you 95% of the work directing incoming email to the right people.

Snippets

We got one other good idea for FogBugz 4.0 from dogfooding it.

As long as FogBugz has had the ability to receive mail, all customer service email sent to Fog Creek goes into our own FogBugz database where anyone can answer it.

Over time I’ve come to notice the occasional customer who thought we were being rude to him in an email message. On closer examination, we weren’t being rude, but the email we sent seemed rude, and it was usually because it was incredibly terse and to-the-point. We decided that instead of sending emails like:

Yep, this is fixed in the latest version.

… we should be sending emails like:

Hi! Thanks for writing to us. I think I know what you’re talking about, and it’s definitely a bug in our product. The good news is: it’s fixed! You can download the latest version by logging onto our online store at https://shop.fogcreek.com with your order ID number and email address, and that should solve this problem once and for all. If you don’t have your order ID number just let me know and I’ll be happy to look it up for you, or call us here at the Fog Creek office at 866-FOG-CREEK.

Please let me know if there’s any other way I can be of assistance!

All the best,

(signature of a real human being)

It’s sort of like translating from English to Japanese. I have never been to Japan but my father, a linguist, once told me the story of the train station in Tokyo, where the announcements were made in Japanese and English. You would hear four or five minutes of nonstop Japanese and then the English translation would be “The train to Osaka is on platform 4.” It seems that in Japanese there is simply no way to say something that simple without cosseting it heavily in a bunch of formal etiquette-stuff. And it turns out the same thing applies to email messages, even in English. The moral of the story is that given two email messages with the same semantic content, the terse one is more likely to come across sounding rude. But given the amount of email correspondence we have to deal with here, we don’t have time to be Emerson on every customer support email.

Thus was born the idea of snippets: canned bits ‘n’ pieces of email that you could insert into an outgoing reply with a few keystrokes. The theory is that you define a snippet like “Please let me know if there’s any other way I can be of assistance” and then assign it to a short code like “2”. Now while your composing your reply, you just hit 2 and then press the backquote key ` and (ding!) it is instantly replaced with the long text it represents right in front of your eyes. It’s really cool and it saves a ton of time and allows us to produce the verbose email replies that are less likely to be misinterpreted. We have built up an extensive library of snippets for all kinds of “faqs” and common parts of email messages.

In tomorrow’s installment of The Road to FogBugz, a look at Thistle, our proprietary ASP to PHP compiler/translator.

It started with a phone call from a customer, back in the old office.

Most of our customers, thankfully, understand that the idea here at Fog Creek is to make shrinkwrapped, off-the-shelf software that you buy for a low price and it already does what you need, and if it doesn’t, well, we’d like to hear about that, but for $99 you’re not getting a customized version of the software, sorry.

Some of our customers still think we’re one of those big enterprise software companies where you call them up, negotiate with a salesperson for three months, and, on the last day of the fiscal quarter, force the salesperson to promise a long list of new features in exchange for a half-million dollar contract.

That’s nice, but we don’t have salespeople and we’re not one of those vendors. Our customers are happy that our price point is low but not all of them quite understand the implications. They ask us to fly out to their headquarters to give a demo of the software to their development team. They send us long spreadsheets with lists of features and ask us to check off the features we support. They even send us RFPs (shudder). RFP stands for “Request for Proposal.” It’s a request by a large company for a custom proposal from a small company. The small company works on the 200 page laser-printed proposal like mad for three weeks and Fedexes it in great expense and at the last minute, where it gets put in the trash because the large company has their favorite vendor who takes them on a helicopter to Atlantic City on junkets involving blackjack and strippers, and who is going to get the contract no matter what, but someone in purchasing for some unexplained reason, maybe he’s bucking for a promotion is insisting that the proposal be opened up to “competitive bidding” and the small company has been chosen as a victim to write up a proposal that has no chance of being accepted just to make the process look a little bit less corrupt, and if you’re a small company, I would recommend that you don’t fall for it and don’t spend any time responding to RFPs unless it’s already understood that you’re going to get the contract.

ANYHOO.

The good thing about such customers is that they give us useful feedback about what features we should be adding to FogBugz. One of our mantras here at Fog Creek is “Listen to your customers, not your competitors.” So when we get calls from potential big customers we listen.

Back then, early in the planning stages of FogBugz 4.0, the call was from a large not-for-profit organization. They were considering using so many copies of FogBugz that they needed a way for department managers to see their own departments’ cases in one place, even when each department managed multiple projects.

The short term solution would have been for us to add a table of departments and make a report based on departments. But we’re an off-the-shelf company, and when we do something, we want to do it in a general way that’s useful to a lot of customers. So we took a little bit more time and designed a more general feature and solved a large class of customer problems with one feature. In addition to customers who wanted things sorted out by department, we had customers who were typical consulting shops and wanted to sort things out by client. Most of these customers wanted privacy features so their clients could access FogBugz without seeing any other clients’ bugs. And many of our customers were small shops with ten or twenty users who couldn’t care less about departments, clients, or privacy features, so whatever we did needed to be implemented in a way that would have zero impact on customers who didn’t care. The last thing we need is for FogBugz to become big and unwieldy because it has a lot of features and fields you don’t care about.

So, OK, it took a little bit of time to deliver this feature to this particular customer. From that phone call to the time we delivered an alpha release to the customer with the new feature about twelve months elapsed. That’s more because we’re still oriented around big major releases every year and a half. I justified the reason for this schedule in the article Picking a Ship Date.

Many of the other features in FogBugz 4.0, which we finally shipped on February 22, were based on customer feedback, too. We kept hearing customers asking how to attach screenshots to a bug. “Oh, it’s easy,” we said. “Alt+Print Screen, run paintbrush, paste, save as a file somewhere, run your browser, go to the FogBugz homepage, hit “New Case,” describe the case, click the file browse button, and find the file you just created.” What could be easier?

Maybe it could be a bit easier.

Finally I thought “how hard could it be to make a little taskbar lint icon thingy that grabs a screenshot and makes a bug out of it?” Not so hard.

I got the Windows version done in The Proverbial Weekend (defined as one weekend to get it working, two weeks to fix bugs, another week to rewrite it to workaround new bugs introduced by a patch to Internet Explorer). Daniel Berlinger knocked out the Macintosh version in REALbasic in a couple of weeks. And now entering a bug that you see on screen is a matter of two clicks.

It’s great.

I’ve found that about 30% of the bugs I enter can be completely described with a screenshot. Here’s a complete bug report I entered on Monday:

The red rectangle comes from a highlight tool built into the screenshot program. The “wha?” I typed was probably overkill. If I had been willing to live with an untitled bug this entire bug report could have been entered with four clicks and a drag (that sounds like a good name for a band: “Four Clicks and a Drag.” Or the Fog Creek development team, now that I think of it.)

The thing is, as far as I know, not that I pay close attention, none of our competitors has this feature.

If one of our competitors think this is cool, they can copy us, but it’ll take them a while, especially if they read my site and bought my line about only shipping every 18 months.

No amount of listening to our competitors would have motivated us to do a screenshot feature. Our customers didn’t think to ask for it, but we did notice that they kept asking for ways to attach screenshots, so that’s what we did.

Come back tomorrow for Part II, in which I talk about dog food.

(Updated 2/6/05)

It’s been about two years since we first installed a server at the Peer 1 colocation facility downtown. That server, a Dell 2650, has patiently handled the load for Joel on Software and Fog Creek gracefully, but there are a few issues that are starting to become more significant as Fog Creek grows and reliability becomes more important. And we need more server horsepower and scalability.

In the past I would have upgraded the system, installed it all, and written an article about it. And then people would have emailed me to suggest better ways to do things, but it would be too late.

So now for the first time ever, I’m going to publish a “live” article here. So far, I haven’t done anything. If you have any better suggestions for how to do things, it’s not too late for me to learn from your experience. As we go along, I might have a few questions for my readers who have experience with this stuff. Thanks to everyone who has emailed me so far!

Step one was figuring out what problems we’re trying to solve. Here’s the list:

• Not enough horsepower to handle the large number of FogBugz online trials we’ll be hosting. FogBugz 4.0 is a couple of weeks away from shipping and that should create a flood of people wanting to take advantage of the free online trial. (By the way, if you want to beat the traffic, apply to join the FogBugz 4,0 beta.)
• Running an email server on Windows allowed us to save hardware, at the cost of buying a commercial email product, IPswitch IMail. We are very close to using up the maximum number of mailboxes we’re entitled to create, and we’d really like to switch to a Linux mail server running qmail or Postfix. The votes seem to be in favor of Postfix.
• Although we have a standby server, a small Dell 750 we stuck in there last summer, we don’t really have a good story for handling failures. Right now we could recover from most kinds of failures in about 24 hours, and we have to take all the sites down for about 15-30 minutes when we need to apply OS patches or upgrade the server. But Fog Creek’s daily revenue has grown to the point where I’m not happy with that. Our new goal is to have no single point of hardware failure be able to bring down any of our sites, to be able to upgrade OSes without bringing everything down, and to be able to recover from very major problems in 15 minutes. This goal is fairly absurd. There will always be single points of failure. The ozone layer, for example. A more accurate goal would be “No single hard drive, power supply, or fan failure takes us out of commission for even 1 second. Any other server failure can be recovered from in 15 minutes. Routine Windows Updates will no longer bring our site down. If something happens to the colo facility, its power supply, or the City of New York, well, we go off the air for a while.”
• And we want to have an architecture in place to be able to scale up our hardware relatively easily to meet new demand.

Step one in this project was finding a good sysadmin to help design and implement this project. One of the great things about Fog Creek in its fifth year is that when we need to do something, we have the money to do it. So instead of me trying to build mission critical server networks using off-the-shelf consumer grade plastic linksys crap, I called up Adam Prato, who I had worked with at Juno, who has years of experience building out these kinds of things. After about an hour briefing he wrote up a very professional plan, sketched out the basic architecture of the new network, and started recommending purchases.

In choosing servers, there’s a lot of benefit to using totally standard, identical parts. It makes it easy to swap things around when you need to scale or when something fails, and things are cheaper by the dozen. In order to avoid going totally broke, we still want reasonably affordable servers, but my requirement of “no single point of failure” means we need servers with dual power supplies, hot-swap RAID, and everything engineered for longetivity. That tends to narrow things down to the Dell PowerEdge 2850, which is the updated version of the 2650 we have now. I looked into the equivalent HP box. It looks like it costs 2-3 times as much for the equivalent thing.

The current plan is to get six servers:

1 and 2: web servers running Windows 2003. Two mirrors, each containing every site we host. We’re using Win2003 instead of Win2000 because mirroring is much easier – the IIS 6.0 metabase is just a big XML file so we should be able to mirror the two machines using something as simple as ROBOCOPY. In fact, I think we can stage all our web sites back at the Fog Creek office and let Robocopy distribute them out to the web servers.

We will have some rudimentary load balancing by using a different IP address for every site that we host, and we can program each server’s NIC to respond to some subset of IP addresses. That way moving a website from one server to the other is a simple matter of changing what IP addresses each server uses. We can scale up by adding more of these servers as we go along, and in the future we can use round-robin DNS or a real load balancer to distribute the load if necessary. If one server dies, the other one is a mirror so it can take up the whole load. Many people wrote in to recommend Windows’ built-in Network Load Balancing. I think this may be overkill for the first generation but it sounds like a good idea in the long run.

3: SQL Server. Because everybody is sharing the same SQL Server, this machine will be slightly different than its peers: it will have the fastest available CPU available and it will have more hard drives plugged in. I heard that it’s a good idea to get the transaction logs onto a different physical drive than the data itself, for better performance, so I think this machine will have two RAID 1 mirror sets (4 hard drives). Most correspondents agreed that this is the best architecture.

4: Mail and everything else server. Linux.

5: Routing server. Probably NetBSD. This machine will have four ethernet ports and will route between the public internet, the direct connection to the Fog Creek office, the “DMZ” containing our public-facing servers, and a private network segment containing the SQL Server machine. Many people suggested OpenBSD for better firewall support (“pf”). I’m out of my league here, and will have to trust Adam to use the right technology. Also having different OSes for #4 and #5 doesn’t make sense so we’ll try to pick one OS for both.

Other people thought that it might be better to use a solid-state hardware firewall/router rather than build our own on a generic Intel server. The advantages of building our own is a lot more flexibility, and somehow I think that the Dual-everything Dell PowerEdge 2850 is going to be a lot more robust than one of those dinky firewall/routers you get at CompUSA. I’ve already had one of those die on me with no explanation…

6: Cold backup. Another machine, identical to the first five, sitting around ready to be used for whatever we need it for or to serve as an emergency replacement.

Each machine will contain:

• A Xeon processor, 800 MHz Front Side Bus, with 1 MB Cache
• SCSI RAID with room for 6 drives. All the systems will start with 2x146GB drives installed in RAID 1 (mirroring) formation, except for the SQL machine which will have 4 drives. Every hard drive in the rack will be a Seagate Cheetah, 146GB SCSI U320, at 10000 rpm. Since they’re hot swappable we can move them around at will and we can keep a couple of spares in the cage. 10K is much cheaper than 15K. Some people suggested using 15K for the SQL Server but I’ve actually heard elsewhere that RPMs aren’t such a big deal for SQL Server. And I really don’t want to have two kinds of hard drive.
• Dual Gigabit NICS. The whole network is going to be gigabit ethernet and we’ll use unmanaged Dell gigabit switches which seem to be priced well below the (NetGear) competition. The managed gigabit switches are more than 3x the price of unmanaged gigabit switches.
• 2 GB RAM
• Dell’s DRAC 4/I remote management card (more about that in a minute).

I called up Dell and got a quote for this whole mess. By the way, if you’re ever buying more than one or two machines, it’s worth actually calling Dell. Somehow you always get a lower price than you would have gotten on the website. I’ve also been told that if you call during the last week of Dell’s fiscal quarter, which I just missed, you get great deals as they spaz out trying to make their numbers for Wall Street in a classic display of American public company mismanagement.

I still have a list of open questions that I need to work on this week. If you have any answers, email is appreciated!

Question 1 – Power Management

Dell’s built-in management card, the DRAC 4/I, has the ability to turn on and off the machine its plugged into, to resolve “stuck” systems, which is nice. But in two years of operating our current server, we’ve learned that the DRAC card itself (an older model, the DRAC 3) is more likely to freeze up than Windows 2000 itself. And there’s no cure to a frozen DRAC card other than a full power cycle.

To address this, out of desperation, we put an APC MasterSwitch in the rack. This is basically a power strip with a web server. You can connect to it using any web browser to turn on or off anything plugged into the back. The MasterSwitch is awesome. It does one thing, and does it well.

The trouble is, this poor MasterSwitch only has 8 outlets and we have 6 servers with two power supplies each. It looks like there’s a company called Sentry that makes something called the Dual Feed Power Tower, which, if I understand correctly, is designed exactly for these servers with dual feeds. Has anyone ever used such a thing? I can’t find a price on the website which always makes me worry. It looks like it’s about $1350. For less money I can get a new APC MasterSwitch Plus plus the “expansion pack” giving me 16 power outlets on two separate boxes thus reducing points of failure.

And another question — according to Dell’s very cool Rack Advisor, my six PowerEdge’s are going to need about 35 amps of power. Yelps. Is that believeable? From what I remember you’re not even allowed to have more than 20 amps on each circuit. So I’m wondering if a single power tower switcheroo thing can handle the load. And speaking of power, sometime this week I need to track down Joe Cooper of Peer 1 networks and find out how much it would cost to upgrade from a 1/8th rack to a 1/2 rack. We’ll be going from 3U today to 14U. I’m hoping that Peer 1 has some way of getting me enough amps to run six servers. Or maybe when Dell says that one of their servers consumes over 5 amps, they mean “fully loaded”, and we won’t really be close to fully loaded. People have told me that Dell is on drugs. Technically the power supplies in these things are 700 watts, but realistically you never draw that much, and so 20 amps should be fine.

Question 2 – Rack Management

As soon as we got the second server, our small backup PowerEdge 750, and put it on top of the old 2650, we had to add RAM to the 2650 (on the bottom), which was a terrible pain in the buttocks. Which made me realize we really need to rack mount all our servers properly. I need to get Peer 1 to tell me what kind of mounting equipment I need to order… Dell gives you a choice of “VersaRails” or “RapidRails.” I’m sure we’ll figure out how to install things, but it would be nice if Dell had a video showing how to rack mount their servers instead of badly written technical documentation with instructions like “Install two 10-32 x 0.5-inch flange-head Phillips screws in the back mounting flange’s top and third-from-top holes to secure the slide assembly to the back vertical rail.” What? VersaRail is for round holes and requires screwing things in. Rapid Rails is for square holes and things just click into place. It’s not that hard, I’m told 🙂

Question 3 – Remote Control

We need to decide between a remote KVM-over-IP system and just using Dell’s built-in DRAC 4/I remote control.

The DRAC has two advantages I know of:

1. It has other nice features like power cycling (when it doesn’t freeze) and the ability to boot your server off of a floppy or CD-ROM which is mounted over the network.
2. It’s a lot cheaper; KVM over IP is pretty expensive.

But I’m worried about the DRAC’s KVM capabilities. With version 3, you could see the computer screen while it was booting up, but after that the remote control capability disappeared for a while until the operating system came up, after which it seemed to use Windows Terminal Services or something like that to remote control the server. This is a little bit to OS-dependent for comfort. I have heard rumors that the new model DRAC 4/I fixes this problem and lets you remote control the server no matter what state the OS is in. Can anyone confirm this? If so we can probably live without an expensive KVM-over-IP solution. The bottom line is that we need to have the DRAC 4/I, because there’s no other way to turn on the server remotely. Power cycling a server that was on will make it come back on, but if you screw up and tell windows to shutdown you can power cycle until next monday and it won’t go on. Given that we have to buy the DRAC 4/Is anyway, I’m hoping that they are good enough and we can live without expensive KVM over IP. If I’m wrong we’ll just add KVM later.

Question 4 – SQL Server on its own Machine

I’ve never tried running SQL Server on a separate physical box than the ASP application that’s using it. I know it works since FogBugz customers do it all the time, and it seems like it should improve performance, but then again, I’m worried about the network bottleneck. We’ll have fast gigabit ethernet and the two machines are only one hop away. Do I need to worry about this? I don’t have to worry about this, it works fine, etc. Many people pointed out that Best Practice is to have a separate network segment/switch for the SQL traffic so it doesn’t compete with  web server traffic. I strongly suspect this best practice arose in the days of 10/100 ethernet, not gigabit ethernet, and that this would be overkill, although if needed it will be easy to retrofit in later since every server has 2 ethernet ports.

Question 5 – RAID And *nix

People tell me that even using Dell-standard RedHat on their own servers with RAID has the problem that there’s no way to find out when a drive has failed, and that the situation is even worse with the BSDs. This could be a major problem. There’s no point in having RAID if you don’t find out when a drive failed. We need to make sure whatever OS we choose for the two Unix boxes supports the Dell Perc Raid Thing.

FAQ

Q: Why Windows for the web servers?

A: FogBugz is written for Windows and ported to Unix. Thus the Windows version is usually about 2 months ahead of the Unix version, and we always want to be running the latest daily builds of our own dogfood. Also, a lot of Fog Creek’s existing applications such as the online store are written for ASP or ASP.NET.

Q: Why SQL Server instead of MySQL?

A: Similar historical reasons, the performance is better, and the Unicode support is far better.

Q: Don’t you like Linux/Unix/MacOSX/NetBSD/BeOS?

A: Yes. I love all operating systems equally. If we were building all this software from scratch I might have taken a different technical approach, but we didn’t, so get off your OS religion high horse for a minute. We’re going to have a mixed environment for the foreseeable future and 90% of FogBugz customers choose to run it on Windows servers. But we will use Unix for network routing, email, DNS, and a few other small things.

Q: Why Windows 2003 instead of Windows 2000?

A: Because IIS 6.0, the web server in Windows 2003, stores all its configuration information in a XML text-file metabase, and supports “XCOPY deployment,” it means we’ll be able to mirror web servers more trivially simply by mirroring file directories. And Windows 2003 has lots of improvements to web server performance.

Q: Why Dell?

A: My experience has been that it’s easier to get a Dell server built and configured exactly the way you want it, and it usually comes out a lot cheaper than the alternatives. At least 2x cheaper than HP! And I don’t like buying through resellers who don’t know what they’re selling. Every time I ever made the mistake of buying something from PC Connection I sentenced myself to weekly annoying sales calls from a clueless sales person who always dramatically screwed up anything I tried to buy from him. When you call Dell you get a nice, cheerful, competent kid in Austin, Texas who laughs at your jokes, no matter how dumb, and actually has the ability to do sophisticated things like read email that you sent him and then actually reply to it!

But that said, Dell has really, really crappy software skills, and their management software sucks big rocks. Every simple application they make is a horrible mess of Shockwave, HTML, Java, and Visual Basic with 35 MB of runtimes just to do some basic system administration thing. Today when I downloaded their otherwise cool Rack Advisor, the download was an EXE — a self-extracting EXE. Running the EXE extracted — wait for it — another EXE. This EXE was the single file installer. Really, really, really crappy software skills. Get some programmers, Dell! HP and IBM are reputed to have much better system management software, not that I’ve ever seen it.

Finally, Dell has a neat feature where you can log on to their website and see a list of every computer you’ve ever bought from them, and I love logging on and seeing that list get longer and longer.

Q: Why not a SAN?

A: There seem to be lots of good reasons to use a storage area network, although I don’t think it’s quite cost-effective for our situation. We’re looking at a total bill in the neighborhood of $20K, and SAN solutions tend to start around that price.

Well.

That took rather longer than expected.

We have, finally, moved, into the new Fog Creek office at 535 8th Avenue, officially ten months after I started pounding the pavement looking for a replacement for my grandmother’s old brownstone where we spent our first few years, working from bedrooms and the garden.

Most software managers know what good office space would be like, and they know they don’t have it, and can’t have it. Office space seems to be the one thing that nobody can get right and nobody can do anything about. There’s a ten year lease, and whenever the company moves the last person anybody asks about how to design the space is the manager of the software team, who finds out what his new veal-fattening pens, uh, cubicle farm is going to be like for the first time on the Monday after the move-in.

Well, it’s my own damn company and I can do something about it, so I did.

Maybe I’m just an architecture queen. I probably pay more attention to my physical surroundings than the average software developer. I might take it too seriously. But there are three reasons I take it so seriously:

• There’s a lot of evidence that the right kind of office space can improve programmer productivity, especially private offices.
• Having drop-dead gorgeous, private, windowed offices makes it a lot easier to recruit the kinds of superstars that produce ten times as much as the merely brilliant software developers. If I have to compete at New York salaries against Bangalore salaries, I’m going to need those superstars, so when people come in for an interview, I need to see jaws on the floor. It’s about drama.
• Hey, this is my job; this is where I spend my days; it’s my time away from my friends and family. It better be nice.

Working with architect Roy Leone, a lot of space (425 r.s.f. per employee), and an enlightened CEO, I set out to create the ultimate software development environment.

Architects use the term “brief” for what we software developers call “system requirements.” Here was the brief I gave Roy.

1. Private offices with doors that close were absolutely required and not open to negotiation.
2. Programmers need lots of power outlets. They should be able to plug new gizmos in at desk height without crawling on the floor.
3. We need to be able to rewire any data lines (phone, LAN, cable TV, alarms, etc.) easily without opening any walls, ever.
4. It should be possible to do pair programming.
5. When you’re working with a monitor all day, you need to rest your eyes by looking at something far away, so monitors should not be up against walls.
6. The office should be a hang out: a pleasant place to spend time. If you’re meeting your friends for dinner after work you should want to meet at the office. As Philip Greenspun bluntly puts it: “Your business success will depend on the extent to which programmers essentially live at your office. For this to be a common choice, your office had better be nicer than the average programmer’s home. There are two ways to achieve this result. One is to hire programmers who live in extremely shabby apartments. The other is to create a nice office.”

Roy did a great job. This is what you pay an architect for. I predict he will become something of a world expert on designing offices for software teams. Here’s how he translated my brief into three dimensional space.

Private Offices. Not only did we get spacious, windowed private offices, but even the common area workstations (for non-developers) are hidden in clever angular alcoves, so everyone gets their own private space without line of sight to anyone else.

The walls between the offices and the workstations are made of high tech, translucent acrylic which glows softly and provides natural light to the interior without reducing privacy.

Power. Every desk has twenty, that’s right, twenty outlets. Four of them are colored orange and have uninterruptible power coming off of a UPS in the server closet, so you don’t need a UPS in every office.

The outlets are right below desk level in a special trough which runs the entire length of the desk, about six inches deep and six inches wide. The trough is a place to hide all your cables neatly and has a handy cover which blends in with the desk.

Wiring. There is a Snake Tray system running near the ceiling from the server room and throughout the office, running through every room. It is completely accessible so if you want to run any kind of (low voltage) cable from point A to point B you can do this neatly. We only moved in Friday, and we’ve already redone the intra-office LAN wiring, in a project which took about half an hour, so the snake tray has already proven itself. Every office has its own 8-port network switch, so you can plug in your laptop and your desktop and your Macintosh and that old computer you keep around to read Joel on Software when your main computer is rebooting to install today’s Windows Update, and still have 3 ports left over (attention math geniuses: no need to email. One port is the uplink.) I sneer at silly building managers who still think that one LAN port per office is about right. For lawyers, maybe.

Pair Programming. When you make typical L-shaped desks many developers set themselves up in the corner. When they need to collaborate temporarily, or pair program, or even just show something to someone on their screen, the second person has to either lean all the way across the desk or look over the first person’s shoulder. To avoid this we designed all the desks to be long and straight so that wherever a software developer sits, there’s always room for another person to pull up a chair and sit next to them.

Resting eyes. Although the desks are up against the walls, there is an interior window in that wall, which cleverly looks across the corner of the next developer’s office and through his window. Because of the rather brilliant layout this doesn’t reduce privacy because even though you have a window onto the next office, it is angled so that from most positions you really only look across a small corner of that room and out its exterior window. The net result is that every office has windows on three sides, two of which look outside, creating the architectural pattern Light on Two Sides of Every Room. This is quite an accomplishment: you try coming up with a scheme to give everyone a corner office in a conventional building. Another reason hiring a great architect was well worth the money.

Hang out. We furnished the office with a kitchenette and a lounge area with sofas and a huge HDTV plasma screen with DVD player. We’re planning a pool table and game console. Private offices means you can listen to music at reasonable volumes without headphones and nobody will care.

Bottom Line it For Me.

The monthly rent for our offices, when fully occupied, will run about $700 per employee. The build-out was done on budget and paid for almost entirely by the landlord. I suspect that $700 per person is on the high side for software developers throughout the world, but if it means we can hire from the 99.9 percentile instead of the 99 percentile, it’ll be worth it.

This may be, possibly, the most off-topic article I’ve ever done here. But I try to write only about things I know about, and, recently, I learned a lot more about commercial real estate than I ever imagined I would need to know.

Old Fog Creek Office, E. 31 st.My company, Fog Creek Software, is relocating. There’s nothing wrong with the brownstone we’re in, it’s just a tiringly long commute. Basically, we’re moving because of William Whyte‘s rule: virtually all corporate relocations involve a move to a location which is closer to the CEO’s home than the old location. Whyte discovered this principle after an extensive study of Fortune 500 companies that left New York City for the suburbs in the 1950s and 1960s. They always had big, complicated Relocation Committees which carefully studied all the options and chose, coincidentally I’m sure, to move to within half a mile of the CEO’s home in Danbury, Connecticut. Whyte also showed that these companies all tanked after the relocation. With, I believe, but one exception, companies that left New York City to be closer to the CEO’s house in Connecticut or Westchester had dismal stock performance compared to companies that stayed in Manhattan.

The dismal stock performance probably came from the fact that when you relocate more than a couple of miles, some employees’ lives would be too disrupted to make the move, so you lose a lot of employees, and all the institutional knowledge, skill, and experience that comes with those employees. While I was working at Viacom one of their companies, Blockbuster, decided to move from Florida to Texas after they hired a new CEO who lived in — Texas! What a coincidence! Only a small portion of the employees made the move. For years and years the business press watched agog as Blockbuster made mistake after inexcusable mistake, re-trying all kinds of ideas that had failed only two years earlier.

But I digress. We’re not losing any employees and we’re just moving across town, to a location that is an easy walk from every subway line in NYC and three mega commuter hubs, bringing us within easy commuting distance for 17,000,000 potential employees.

Temporary Space and Incubators

Step one in finding an office in New York: allow nine months. I am not kidding about this. It will take about two months to find the space, a month to negotiate, two months to sign the lease, a month for your architect to design the build out and three months for construction. If you run out of time on your old lease, chances are (look closely) you have a clause in there which doubles or triples your rent if you stay past the end. This is called the overhang and next time you negotiate a lease, you’ll remember to negotiate a lower overhang.

Nine months. You may think you can shorten that by taking pre-built space. The problem: there’s hardly anything pre-built in the office market in New York. Almost everything you see is going to be either completely raw, or it’s going to be a weird warren of dark rabbit offices with inexplicable bars between certain rooms, filthy carpet, and disgusting yellow-stained acoustic tiles.

What there is, if you have to move tomorrow, is temporary executive suites. These are the things you see advertised in the newspaper that say “Move Tomorrow! Plans from $100 a month!” What the ad doesn’t say is that the $100 a month is for a mailbox only, with everything else a la carte. If you really want an office in one of these executive suites you’re going to pay through the nose: about five to ten times as much as comparable office space. It’s like living in a hotel instead of living in an apartment, or flying Concorde to London. Yeah, it’s convenient, but is it really $10,000 a month convenient? Still, all the executive suites in the city are full, usually an honor roll of venture-capital backed companies working hard on keeping their cash burn up to respectable levels to impress the investors. This is because normal landlords hate venture capital funded companies. The few landlords in New York that rented to VC startups during the dotcom boom have learned never to do that again. Here’s why: venture funded companies only have enough cash to last a few months or at most a year, after which they have to raise another round. If they don’t raise the round, they close. Probably 80% of these companies close within a year. The way most leases are written, a landlord literally loses money if the tenant leaves after a year. If you’re venture funded, and your VC won’t sign a five year lease for you, you may be stuck paying the Rich Tax, paying $200 per foot (per year) to be in executive suites when the market rent is $20 – $30.

Another option is the so-called incubators. In New York City, we have TechSpace, EEmerge, Kickstart, and probably a few others. I am sort of surprised these places survived the dotcom bust, but they did, and they’ve lost their arrogance. They no longer act like VCs, they don’t care about your business plan any more, and they don’t want 5% of your pathetic worthless stock, they just want to get paid. They won’t even kick you out after 3 months any more.

Incubators are a terrible place for programmers, because they took the baby chick metaphor a bit too far. The scientific geniuses that built these places actually think that the closer you shove people together the warmer they will be and thus the faster they will grow. What you get at the incubators is a room the size of your bedroom with desks around the outer wall where you can cram 16 people, all for about three thousand bucks a month. Oh, you wanted a window? Make it four thousand. Incubators are a smidgen cheaper, per square foot, than the executive office space, but they are just not an acceptable long-term solution.

Normal Office Space

A quick survey of normal office space.

First, it’s usually not built, and if it is, somebody has probably been using it. If the walls are configured to your liking, you may be able to paint and recarpet it (about $7 per square foot) and move in. Otherwise you’re going to have to do construction. Plan on this costing $50 per square foot. So if it’s a 3000 square foot office, expect to spend $150,000 on the construction. I know, this sounds like a lot. While you are looking for offices, brokers will tell you that you can construct an office for less than $50 a square foot. They are lying to make the deal. Do not fall for it. The only way to pay less is if a lot of the work is already done (on our office, the demolition, ceiling, sprinklers, air conditioner, bathrooms, and finished floors were all done, so we’re hoping to spend $25/ft on construction.)

Update, Feb. 2006: Construction costs are up a lot in the city. Plan on $75/sq ft for a very minimal buildout. I’m serious. We did construction for about $30 in 2003, but that was with an incompetant contractor who did terrible work, took forever, and lost his shirt on the deal because he had underbid.

In today’s market, you can usually get the landlord to pay for some of this, probably $10 to $25 a foot. They will want receipts and stuff and they will only want to pay for walls and air conditioning ducts, not Aerons and plasma TV screens. Some landlords will also offer to do the construction themselves for you. If you take them up on it, you’ll get the cheapest possible construction and it won’t be very nicely designed, so you may be better off hiring your own architect and contractor.

In the office space market in the US, offices are categorized as Class A, B, or C. Class A means new, top-of-the-line construction. In New York, that means one thing only: modern steel and glass high rises. Anything else is not Class A. A classic old skyscraper from 1913 designed by Cass Gilbert with incredible detailing, beautiful gargoyles, a complete renovation from top to bottom and everything is spotless and new is still not a Class A building, even if it’s nicer than most Class A buildings.

Class B and Class C don’t mean anything, there’s no formal definition, so nobody bothers to use these terms in the city any more. There’s Class A and there’s Everything Else.

For the purposes of high tech startups, unless you need a whole floor of a skyscraper, forget about Class A. It’s not that you can’t afford it, it’s that they are not going to subdivide floors for you. These buildings are leased 10 floors at a time.

That leaves you with a wide variety of old buildings, often loft buildings, in various stages of renovation. The best ones have been recently renovated, the elevators are new, everything is spick-and-span and the lobby is nice if not glorious. The worst ones have a spanish deli as the lobby, the (one) elevator is manual and operated by a scary old man, and there are still a lot of clothing factories in the building operating heavy machinery that goes bonk bonk bonk all day long. Almost all of these buildings are family owned, although they might be managed by one of the large real estate management companies like Newmark or GVA Williams.

What If I Really, Really Don’t Want to Build?

There are a few buildings which cater to small tenants where the offices are basically built out and at most you’ll need paint and carpet. The Graybar Building near Grand Central, the Fisk Building near Columbus Circle, and the hugely unpopular Empire State Building. Graybar and Fisk always have spaces available on short notice at “market rents,” about 50% – 100% more than you’ll pay for raw space in the Garment District or Wall Street but reasonable for their neighborhoods. Graybar is a funny ghost town building, right on top of Grand Central; it’s full of offices of just-retired lawyers who live in Connecticut and can’t quite let go, so they keep an office in the Graybar building, but then they never really go there.

Do You Need A Broker?

In the commercial market, brokers are paid by the landlord. Usually something like 6% of the total value of the lease over its entire term. This is a lot of money for a little bit of work. So as soon as you let it be known that you are looking for office space, you will have brokers falling all over you.

You have the option of using a tenant’s broker. A good tenant’s broker does all the legwork for you, looking at lots of spaces to find one and then showing you only the best options to choose from. The trouble is, there are hardly any good tenant’s brokers. Most of them are just too lazy. The will tell you to meet them at Starbucks at 10:00 am, and they’ll bring a printout they made at 9:45 that you could have gotten online (more about that in a moment), and they’ll make you walk around with them while they get their act together and discover that these spaces don’t really exist.

Tenant’s brokers will tell you that they have the inside track on office space that isn’t listed, especially sublets. The best ones will, but only the very best ones. Most of them are just making the same printouts you can get yourself.

Bottom line: you don’t really need a tenant’s broker to show you spaces. OK, the other thing the tenant’s brokers will tell you is that they represent you, not the landlord, so they will have your interests in mind. This is vaguely true, but they’re still getting paid by the landlord, and they still make more money if you overpay, so it’s hard to trust them. If you’re worried about landlord sharks taking advantage of you, stop worrying about that. Why? Because you’ll have a good lawyer on your side when you sign the lease. You need a lawyer to review your lease anyway, that’s your real protection from the landlord, not a broker who gets paid by landlords.

Are there disadvantages to having a tenant’s broker? There are two. First, you may have to sign an exclusive. In today’s market, with vacancies skyrocketing and no new tenants, you should flatly refuse. There’s no reason to sign an exclusive. Just promise the broker that if you take a space which they showed you first, you will see to it that they get their commission.

The other disadvantage is that if a tenant’s broker finds you a space and you take it, they will be splitting the commission between the tenant’s broker and the landlord’s broker. That means the landlord’s broker only gets half the commission. That means, all else being equal, that the landlord’s broker would rather do those deals where they get the whole commission, when they have a choice. That means that if you’re competing for space with another potential tenant, and that tenant has his own broker and you don’t, then the landlord’s broker is going to like you much more… twice as much, in fact. Although theoretically it’s up to the owner of the building to decide whom to rent to, you can bet that the landlord’s broker will find ways to make the owner like you more.

Update, Feb. 2006: I’ve been told that many landlord’s brokers are salaried employees on the staff of the building management company who couldn’t care less and don’t really work on commission. So for larger space (10,000 square feet or more) it probably won’t hurt to hire a good tenant’s broker and sign a three-month exclusive, but make sure you’ve got a really good broker who will do the footwork to earn their commission.

How Do I Find a Space Without a Broker?

Easy as pie.

First, do searches on two websites, which overlap some but not entirely in their listings: CityFeet and MrOfficeSpace.

Next, check the Sunday New York Times. (Not online. The online version just redirects you to CityFeet, which has a superset of the print classifieds). Circle everything that looks appropriate.

If you want cheaper, artsy space, check the Village Voice.

Avoid two things: ads that sound like executive office spaces, and ads that are just pitches for brokers disguised as listings. You can identify these because they don’t give any details about the space, like the address. If the address is not in the listing, don’t bother responding, it’s almost certainly a broker who is fishing for potential tenants. He’ll tell you to meet him at Starbucks at 10:00 am where he’ll show up with exactly the same list you printed out from MrOfficeSpace.

Next step: do not call the broker in the listing. Yes, they can show you the space, but you don’t need them yet. First go to the building in the listing and see if you like the building. Check out the lobby. Read the list of tenants to see if they’re mostly software companies, architects, and graphic designers, or if they’re mostly clothing factories, importers, and methadone clinics. Check out the neighborhood. In the bad buildings, people get into the elevators talking to themselves. In the good buildings, people appear to get into the elevators talking to themselves, but they have a tiny earphone, so it’s totally different, they’re just on the phone.

Like the building? Here’s the next step, for which you should be reasonably well dressed. Go into the building’s service entrance and ask to speak to the super. Tell the super you’re looking for office space and you want to see the space in his building. He will show it to you. In better buildings, he may ask for a business card. That’s all there is to it.

Now, if you like the space, call the broker.

“What would have happened,” you may ask, “if I had just called the number in the ad?” Well, nothing terrible. The broker would have met you in the building lobby, and then he would have asked to see the super, and the super would have shown you the space. You see, the super has the keys and knows how to drive that cool manual elevator that goes to the floors where the fancy automatic elevators don’t stop because there are no tenants there.

At this point, if you don’t like the space, the broker will offer to show you a zillion other spaces. If he’s smart he’ll have the MrOfficeSpace printout that you also have. It doesn’t hurt to walk around with him and look at spaces, he may know the neighborhood and be able to give you the gossip. It’s worth spending some time talking to all the brokers you meet. Ask them what buildings have nice owners, and what buildings have nasty owners. They will lie a little bit about the buildings in which they have an interest, but if you keep asking and average the results you’ll eventually learn the reputations of the landlords in the neighborhood.

How Do I Know If I Like The Space?

I kept looking at spaces, not particularly knowing if I liked anything or not, and one nice broker said to me: “Don’t worry. When you find the right place for you, you’ll know it.” And he was right. The next day I walked into a place that was beautiful: natural light on three sides, a great renovated building, beautiful new wood floors; instead of taking notes I just wrote “incredibobble” in my notebook. That’s the place you want. If it doesn’t happen keep looking.

Gotchas to watch out for:

• If there are windows, figure out what might happen that would block your views. If the building is next to an empty lot or a “taxpayer” (one story building), as soon as the economy gets better, a building might go up there and it might block your view.
• Check out the bathrooms. Some buildings have really gross bathrooms and people forget to look at them.
• Sit quietly in the space for a few minutes and listen to all the noises. Are there factories nearby? Noisy people upstairs banging around? A noise you don’t notice when you’re walking around talking to a broker may be torture when you’re trying to concentrate on writing code.
• Make sure the building has 24/7 access. Some buildings in New York actually close.
• How are you going to get air conditioning? Is there something in place or will you have to install it? When you install it will it block your only window?
• Are there enough elevators? Older buildings with only one or two elevators can mean really long, irritating waits.

How Big A Space Do I Need?

Space is measured in RSF: rentable square feet. This is a number that bears only a fleeting relationship to the actual size of the space. I have seen spaces that are 1000 RSF which are the same size as spaces that are 3000 RSF.

How could this be, you ask?

Well, you don’t want to know. Officially you get RSF by taking the total size of the space and multiplying it by the building’s loss factor. That’s a made up number which they multiply the size of the space by, just because they can get away with it. It’s supposed to account for all the shared areas, like the basement, the elevator shafts (counted once for each floor of course), the exterior walls, the interior walls, the bathrooms, the lobby, the Starbucks next to the lobby, the staircases, the fire escapes, and the owner’s house in Queens including pool and garden.

Meaning — RSF is pretty useless. When they quote a price, it will always be quoted in dollars per square foot per year. Multiply by the size and divide by 12 to get your monthly rent, and this is the only figure you care about. Sign a lease based on the cost, not the square feet, or they’ll “recalculate” their loss factor the next year and raise your rent.

This makes it sort of hard to search for office space, and it’s one reason you have to look at lots of spaces (and bring a tape measure.) If you calculated that you’re going to need about 2000 square feet, look for places advertised between 1000 and 3000.

Hidden Charges

Find out about all the charges you’re going to have to pay. There are a couple of ways New York buildings especially like to stick it to you.

Some buildings don’t have separate electric meters for each space. Since they don’t know how much electricity you’re using, they just have to charge you based on the assumption that you are operating a 100,000 watt radio station with Times Square lighting. They’ll charge you something like $3 per square foot per year which is way more than you would spend if you had your own meter.

Some buildings make up things and put them in the lease. “Tenant shall pay $200 per month to maintain the decorative steel trim on the building,” my lease said. Hmm. Damn building doesn’t even have a steel trim. I made them take it out.

Escalations are another big issue. If the rent is x for the first year, it’s going to be x times the escalation in the second year, and so on. This amount should be agreed upon and not based on some weird fabricated thing like “tenant’s prorated share of the owner’s increased expenses from operating the building.” I don’t care about the owner’s expenses and I’m not paying for the owner to pass along the cost of his 2003 Buick LeSabre to me. Some buildings have a uniquely Manhattan corruption called “Porter’s Wage.” This means that every year, when they renegotiate the contract with the Porter’s Union, the percentage that those salaries goes up by is the percentage escalation that the tenants pay in rent. The theory was that a porter’s wage was correlated with the overall inflation in the cost of operating a building, but in reality buildings don’t have porters any more, and for some reason the porters they don’t have would get really huge salaries, if they had jobs, but they don’t, and the tenants are still paying for it.

Another thing the landlord will want to pass on to you is your share of the increase in property taxes. This is reasonably standard. They will even bill you for any money they spent “lobbying” (paying bribes) to get their property taxes reduced. Slimy but common.

How Do I Negotiate The Lease?

Theoretically, everything is negotiable. Practically, you’re going to negotiate these big items:

• monthly rent.
• months of free rent at the beginning (these days it’s not uncommon to get one month for each year of the lease, and even a couple more months while construction is going on)
• landlord’s contribution to buildout
• length of the lease

For the purpose of negotiations, there are a few things you can pretend to care about which you can later swap to get a better deal:

• pretend that you want 100% of the profit from a sublet. They won’t give it to you, they’ll give you 50%, which is actually better for you because it gives them an incentive to allow the sublet in the first place, and you can swap this for something else.
• pretend that you don’t want to pay a security deposit.
• pretend that you want zero overhang, so your rent doesn’t go up if you overstay your lease.

You may think you want to sign as short a lease as possible, because it “leaves your options open.” Actually you’re probably better off with a longer lease, for lots of reasons.

• Landlords like long leases, so they’ll like you more if you sign a long lease, and you can get a better deal.
• Rents are super low now. If rents go up, your lease becomes worth cash money.
• If you grow out of the space, the landlord will let you out of the lease if you take another space elsewhere from the same landlord.
• If you go bankrupt or close, the landlord is not really going to be able to get anything out of you. If you’re incorporated and the corporation signs the lease, you’re not personally liable. At worst, the landlord will ask the company’s officers to personally sign a good guy guarantee. This says that you will be personally responsible for the rent but only until the premises are vacated.
• If you need less space, you can probably sublet. You might lose some money on the sublet but you’ll probably make money. The landlord will help himself to half of your net profits from subletting.
• You’ll get more free months at the beginning with a longer lease.
• You’ll be able to negotiate more money for construction with a longer lease.
• The broker will get more money, so he’ll be more eager to make the deal happen.

All in all, there’s no reason to be unduly scared of long leases. You can start by asking for a three or five year lease, agree on the months of free rent and construction money, and then ask them how much more free rent and money they’ll give you if you take a seven year lease. (Logically, this should be proportional — doubling the length of the lease should double the free stuff).

There are thousands of good books on negotiating in general and even a bunch of books on negotiating leases. Read two or three if you’ve never done this before. Never, ever go into negotiations unless you have a good alternative. You should have two or three options you like and want before you try to negotiate.

After you negotiate the big items, get a proposed lease from the landlord and let your lawyer negotiate all the little details to his lawyerly heart’s content.

Do I Need a Lawyer?

Absolutely. First of all, the errors-and-omissions insurance you carry (you do have errors and omissions insurance, right?) requires you to have a lawyer review any contracts you sign. But it’s a great idea anyway.

I found a lawyer who works for another landlord. He spends most of his days in court trying to evict tenants. I thought he would be perfect to protect me from the likes of — him. And he was. As he was reading through a lease, he said to me, “OK, this paragraph, I really like this clause, I think I’m going to steal it and use it in my own contracts.” Pause. He reaches for a red pen and crosses out the paragraph. “But you’re not signing it.” This guy was great. He knew that the standard phrase in the lease where it talks about leaving the space in “broom clean condition” was a classic way to avoid returning the deposit. “Sure, it was clean,” the owner says, “but it wasn’t broom clean. Prove to me that it was broom clean.” We crossed out “broom” and wrote “reasonably.” You Need A Good Lawyer For This.

Here’s another reason you need a lawyer. The lawyer can say to the landlord, “Whaddaya mean if we’re 3 days late on the rent we get evicted? We want the right to be 15 days late!” When your lawyer says this, no offense is taken. He’s just the lawyer. But if you, the tenant, try to say it, the landlord will say, “What, are you planning to be late on the rent?”

Oh, It’s All Too Hard!

I know. I know. And it has nothing to do with writing software. But as soon as you sign the lease, you can hire an architect and start talking about the fun stuff, like where the pool table goes.

Other Links

TenantWise has a lot of great articles about all this stuff, although you have to register to read it all, it’s worth it.

Crains New York Business covers the commercial real estate beat like no other publication. Read it for a few months before you need office space, especially pay attention to the boring stories about deals, so you know what the market is like and negotiate from a position of knowledge.

Read the last few months of news at CityFeet, too.

Discuss your experience on the Joel on Software New Yorker’s Forum.