2002/04/30

Tom DeMarcoToday at lunch I finally got a chance to do something I’ve been waiting for ten years to do: I sat down with Tom DeMarco to talk about office space. I’ve been a strong believer in private offices for developers, something which Tom advocates in his bestseller Peopleware. I asked him about how pair programming can be reconciled with private offices; I imagined that a buzzing room full of developers working in pairs would be too distracting an environment to get into flow and stay there. Tom said that groups of people who work in the same style on the same kinds of tasks will not distract each other. Programmers working on the same project would not distract each other; a room full of sales reps who spend most of the day on the phone don’t distract each other; etc. You get in trouble, though, when you put a programmer in a room with the sales reps. We generally agreed that if you keep the noise level down and keep the outside interruptions away, it’s OK to have an open space floorplan for your development team.

That said, if you’re doing pair programming, you’re not going to spend more than 3-4 hours a day working with your code buddy. For all the other things you do (phone conversations, design discussions at the whiteboard, etc.) it’s nice to have private offices. And when you’re trying to recruit superstar programmers they will expect a plush private office. (Just try and get Tom Cruise to make your movie if he doesn’t get his own dressing room.) So now I’m envisioning the ideal development interior plan as one where programmers have private offices but they meet in labs with their buddies to get production programming done. No email, phones, or non-programmers would be allowed in the labs. All the non-programming work you do — design, negotiation, scheduling, reading, thinking, and Tetris — could be done in your private office.

World’s Worst Misinterpretation of The Mythical Man Month

When I read this article by Scott Rosenberg for Salon, I thought it was a complete misapplication of the Mythical Man Month. Reviewing a large body of existing code to find security problems is easily divided up among huge teams of people, and should not suffer from any of the Mythical Man Month problems. The article is just flat-out wrong. The nice thing about being at the Cutter Summit is that I can ask Mark Seiden, one of the world’s foremost experts on computer security, if I’m crazy. “You’re right,” he said, “it doesn’t apply. There are no interdependencies to speak of.”

2002/04/28

Ah good; free high speed Internet access in my hotel room.

Extreme Programming

One of the things that I thought made Extreme Programming inappropriate for shrinkwrapped software was the fact that XP calls for frequent iterations, which is usually interpreted to mean frequent releases. With shrinkwrapped mass market software, you just can’t do that. Tech support would become completely impossible if some customers were using Windows May 15, 1998 while other customers had Windows May 16th (Second Release), 1998. There are similar problems with embedded software. You really can’t rerelease the embedded software that runs your BMW engine every day or every week.

Picture of Kent BeckTalking to Kent Beck about this today, he pointed out that the simple solution is to do frequent iterations but only release on a longer schedule. Thinking more about this, I think we might move to a model at Fog Creek where a select group of volunteer customers receive very frequent releases in exchange for providing very frequent feedback, while the rest of the world gets far less frequent releases. The nice thing about this is that we can design something, implement it, and realize that the design was wrong and then be able to change it without having to maintain backwards compatibility with the old, “wrong” design (CityDesk customers following the whole URL naming convention switcheroo will understand what I’m talking about, here.)

Some other notable Beck quotes: “The fundamental project management question is: we’re halfway through, are we halfway done?”

And on the topic of rolling out XP to a large organization: “It’s not divide and conquer, it’s conquer and divide.” Get your XP teams working right first, then split them and add new people to each half. Building a consistent culture for code development requires that everyone is on the same page. Add people too fast, and you’ll find yourself with lots of different cultures, which creates conflict, inconsistency, and problems at the edges.

2002/04/27

The New York Times: “Things are so needlessly complex because featuritis sells products,” Dr. Tenner said. “People buy them for a feeling of control, then complain that they are so hard to manage. But show them something simple and rugged, and most of them will call it boring.”

Featuritis sells products, but choices reduce usability. The really great designs are the ones that appear to eliminate a choice. You know you’re doing your job as a designer when you figure out a way to take a complicated feature and make it simpler. Windows NT has a complete two-way folder synchronization feature hidden behind a single menu checkbox “Make Available Offline.” Sometime around 1992, the Excel developers managed to reduce the sorting feature to a single click — this actually took a lot of work internally, because it has to figure out which column to sort, where the table really is, and whether there are row headings that shouldn’t be sorted. Today I’m writing JavaScript client code to make bug editing a little bit smarter in FogBUGZ, which will completely eliminate the Move command for moving bugs from project to project (because you’ll be able to just edit the bug to change the project). This takes more code and more validation and I have to make sure it works even if the browser doesn’t support JavaScript. It usually takes a lot more code to make a simpler interface.

2002/04/25

There’s a lot of SOAP backlash these days. For example Paul Prescod is moping about how SOAP is so much more complicated and less efficient than what he calls XML/HTTP/URI. Everything he says is factually correct, but his conclusion is a bit off the mark.

Yes, SOAP is more complicated than URIs with XML responses, but only if you are the kind of person who hand codes your web services calls. The point of SOAP is that you use higher level languages to call it and to implement it and then it’s easier and better and typesafe and all that good stuff.

Many things in SOAP are there to make it easier. WSDL is there so that I can type “google” followed by a dot and immediately get a little intellisense popup telling me what methods I can call. Then I can choose doGoogleSearch from the popup list and get intellisense again prompting me for all the arguments, and I can get little wavy red underlines in my editor if I pass a string where a long was expected.

This is all stuff that makes the API easier to use. The claim that SOAP is bad because the wire format is ugly and hard is like claiming nobody should use Pentiums because their instruction sets are so much more complicated than the instruction set for the 8086. Yeah, it’s true, but we have compilers that take care of that, and lo and behold, all those complicated hard things like rings and virtual memory actually make it easier for the many who use compilers at the expense of the few who write compilers.

I think most of the criticism of SOAP can be traced back to a few grumpy language vendors who don’t feel like doing the hard work to support it (especially WSDL), or are still living in the archaic land of variants, not intellisense, so they can’t really figure out how to do WSDL. Ten years ago, we used variants extensively in our scripting languages (ExcelVBA, JavaScript, perl) because the kinds of novice programmers who were getting their feet wet didn’t understand types, and even experienced programmers knocking off quick-and-dirty scripts didn’t want to bother with types. Today, when you have a strongly typed language, a good type library (IDL, WSDL, Java reflection) documenting your classes, and an IDE that knows how to show you that type library as you code (i.e. Intellisense), it’s even easier to use types than variants, so the trend is away from variant-bound scripting languages (VBScript, PHP, perl) towards typed code (C#, JSP).

Prescod’s most valuable point is the performance issue, which happens to matter for Google because they have thousands of servers, but it doesn’t matter as much for 99.99% of the web services that are going to be deployed in the real world. (Assuming the post-Web-Services-Hype-backlash doesn’t kill us all before then…)

The real problem with SOAP is that it’s a completely inadequate remoting system. It doesn’t have events (which makes it useless for large classes of applications). It doesn’t support references (ditto). It has about 10 years to go before it gets to where CORBA and DCOM were 10 years ago. And we’re supposed to be all excited about this because we can sneak through the firewall. Gee, I wonder what the firewall vendors are working on these days? As soon as they’re done, we’re even further back than we started.

Actually…

The real problem with SOAP is that it’s the poster boy for Web Services, which are just the next step in a long line of technologies that has been way overhyped by bored trade rag editors and conference organizers and web pundits. You know that something is overhyped when every stupid VC funded startup feels compelled to change their strategy to get behind the latest flavor of the day. Look at poor KnowNow; with all the hype about Web Services they confused their already muddled and misunderstood message by trying to jump on that bandwagon … but the Hype Kingmakers at InfoWorld were not falling for it! No Sirree! KnowNow was officially scolded for not being real Web Services.

Attention, young startups: the trouble with jumping on InfoWorld’s fake hype bandwagons is that by the time you get there with real products, you’ll be smacked in the face by the Tyrannosaurus Rex tail backlash. It’s like the VCs who watched the Lycos IPO and suddenly decided to fund portals. By the time the products came to market, Lycos was over and long gone and everybody was ridiculing portals.

Just watch, I predict within three weeks InfoWorld runs out of ideas for Web Services stories and has to run a big spread on how Web Services were “overhyped” and now CTOs are “revolting” against “vaporware.”

2002/04/19

I’ve been having a lot of fun developing The Fog Creek Control Panel with ASP.Net. It’s a web page that allows us to track critical business data in real time. I can see things like how many people are registering the CityDesk starter edition, how many affiliate applications need to be processed, how many bug reports have arrived from customers (2 so far today), and so on. This morning I added two features: one gets detailed order information for any purchase in our database, and the other lists open purchase orders people have placed with us. Elapsed time to implement both features: 20 minutes.

To be fair not everything about .NET is peaches and cream. Here are a few steps backwards I’ve noticed so far:

  • the IDE is amazingly sluggish on my PIII/600 laptop. (Speaking of which, why the heck can’t I get a Thinkpad X-series with a 1.7 GHz P4M already? Does anybody know of any ultralight notebooks with this chip?)
  • since destructors aren’t called when an object drops out of scope, if you have a SqlConnection object, you positively have to remember to close it because you can’t wait for the garbage collector to come around. I suppose this limitation of GCs has been argued to death when Java came around, and it’s not a big deal, but it is a step backwards
  • that 20 MB runtime is a real showstopper. Yes, this is an advantage to Delphi. I’m starting to suspect that Microsoft doesn’t care — they are perfectly happy to have a development environment that is excellent for internal/corporate applications, where they don’t compete, but borderline useless for shipping, commercial, shrinkwrap, where they do compete.

 

2002/04/12

Whitelist spam filters are probably the last hope for dealing with spam in a meaningful way, and I can tell that people are starting to use them — the last notification email I sent out resulted in a few replies saying “please respond to this message with blahblah in the subject line if you want me to read it” which is what the whitelist programs bounce back.

I can’t really process all of these bounces manually; my mailing lists are managed by a professional service provider (Whatcounts; recommended). The moral of the story is that if you have a whitelist filter, please make sure that *@whatcounts.com is allowed or you won’t get the mailing list. Also, all mailing list messages will always contain [JoelOnSoftware] in the subject line.

Open Source and Usability

Matthew Thomas, a Mozilla contributor, wrote something interesting about why free software usability tends to suck.

Coupons are Fun

We just implemented coupons for our web shop. Nifty. Here’s a treat for the weekend: enter MISSPIGGY in the coupon box to get CityDesk Home Edition for $59 instead of $79. Expires April 15th.

(The serious point of coupons is that they let you try different offers on different people and figure out which ones are the most profitable. For example we could send 1000 people a coupon for $20 off, and 1000 people a coupon for $40 off, and compare the response rates).

2002/04/11

According to the New York Times, Microsoft has pretty much abandoned their plans to centralize everyone’s personal data.

The plan, a version of Passport on steroids called Hailstorm, never had a chance, for three reasons. First and foremost, Microsoft could never get any other company to go along with it. Partially this is because nobody trusts Microsoft any more; more significantly, it’s because there’s no real benefit to the other companies. Second, consumers weren’t about to trust Microsoft with all their juicy personal data. There was just too much of an uproar. And finally, as in now becomes clear, this was Rick Belluzzo‘s pet project, and, as anyone could have told Rick, Microsoft doesn’t really like executives brought in from outside, even though they keep bringing them in.

Moving to .Net

I wrote up our current thoughts about moving to .NET at Fog Creek.

2002/04/10

Guerrilla Marketing

For reasons that are explained in today’s ship date article, my company’s current marketing efforts are focused on low profile, word-of-mouth marketing, not Superbowl ads.

As a company founded by programmers, it’s no surprise that a lot of what we do for sales and marketing is done by writing code, not by making phone calls or hiring ad agencies! In that spirit Michael has finished setting up an affiliate program for Fog Creek Software. A little ASP code, some database magic, three thousand pages of lawyer-approved legalese, and hey presto! an automatic sales program that costs very little.

If you have a web site, you can link to CityDesk or FogBUGZ and earn 15% any time someone buys our software through your link. We also decided to actually allow you to link to Joel on Software and TechInterview with the affiliate program. A lot of people hear about our software through these sites. So web sites that would link to JoelOnSoftware anyway can make money if anyone they refer eventually buys software from us (it works using a 30-day cookie).

To get the new program off to a good start, I’m going to raise the commission to 20% for anyone who signs up by Friday. And if you make your first sale before May 1st, we’ll boost your commission to 25% permanently. You could make almost $500 if someone you refer to us buys a single FogBUGZ site license.

Today’s Article: Picking Ship Dates

One of the best reasons to make a detailed schedule is because it gives you an excuse to cut features. If there’s no way to make the ship date and implement Bob’s Singalong MP3 Chat feature, it’s easy to cut that feature without making Bob feel bad.

But — how do you pick a ship date?

 

Picking a Ship Date

One of the best reasons to make a detailed schedule is because it gives you an excuse to cut features. If there’s no way to make the ship date and implement Bob’s Singalong MP3 Chat feature, it’s easy to cut that feature without making Bob feel bad.

So my basic rule for software release cycles is:

  1. Set a ship date, which might as well be arbitrary
  2. Make a list of features and sort them out by priority
  3. Cut low-priority features every time you slip so as to make the date.

Box FactoryIf you do this well, you’ll soon discover that you don’t regret the features that you cut. They’re usually kind of dumb. If they were so important, you can do them next time. It’s like editing. If you ever want to write a brilliant 750 word essay, start by writing a 1500 word essay and then edit.

One way to screw this up, by the way, is forgetting to do features in order of priority. If you aren’t careful, your programmers are going to do features in order of fun, and you won’t be able to ship or cut features because all the programmers were busy writing Karaoke Recalc before they even got the menus to work, and now here it is, six months after the day you wanted to ship, and you have one hell of an easter egg but no functionality.

So the obvious question is: how do you pick a ship date?

It’s conceivable that you have outside constraints. The stock market is switching from fractions to decimals on such-and-such a date, and if you haven’t got the new software ready, your firm will be forced out of business and you personally will be taken out behind the loading docks and shot in the head. Or maybe a new version of the Linux kernel is coming out soon with yet another all-new system to implement packet filtering; all your customers are getting it; and your existing application won’t run on it. OK, for those people, your ship date is easy to figure out. You can stop reading this article now. Go cook a nice dinner for your loved ones.

Bye, now!

But how should the rest of us pick a ship date?

There are three approaches you could take.

  1. Frequent Small Releases. This is the Extreme Programming approach, and is most appropriate for small-team projects with a small number of customers, such as in-house IT development.
  2. Every 12 to 18 months. This is typical of shrinkwrapped software products, desktop applications, etc., where you have larger teams and thousands or millions of customers.
  3. Every 3-5 years. This is typical of huge software systems and platforms that are worlds unto themselves. Operating systems, .Net, Oracle, and for some reason Mozilla fall into this category. They often have thousands of developers (VS.Net had 50 people on the installer team) and enormously complex interactions to other shipping software which can’t be allowed to break.

Here are some of the things you need to think about when deciding how often to release software.

Short releases get customer feedback fast. Sometimes the best way to work with a customer is to show them code, let them try it out, and immediately incorporate their feedback into a build which you give them the very next day. You won’t waste a year developing a complicated system with lots of features that nobody uses, because you’ll be so busy doing things that customers are requesting right now. If you have a small number of customers, prefer frequent small releases. The size of each release should be the minimum chunk that does something useful.

Several years ago I was assigned to develop a web content management system for MTV. The requirements called for a database-backed system with templates, and a complete workflow system that allowed unpaid MTV stringers at colleges across the country to input information about clubs, record stores, radio stations, and concerts. “How are you building the site now?” I asked.

“Oh, we just do it all manually with BBEdit,” they told me. “Sure, there are thousands of pages, but BBEdit has a really good global-find-and-replace function…”

I figured the whole system would take six months to deliver. “But let me suggest something else. Let’s get the templating stuff working first. I can get you that in 3 months and it will save tons of manual work right away. Once that’s working we’ll start in on the workflow component; in the meantime you can continue to do workflow with email.”

They agreed. It sounded like a great idea. Guess what? Once I delivered the templating feature, they realized that they didn’t really need workflow that much. And the templating turned out to be useful for lots of other web sites which didn’t need workflow, either. So we never built workflow, saving three months which I used to enhance the templating feature which turned out to be more useful.

Jewish Quarter

Some types of customers don’t appreciate being “guinea pigs” in this fashion. Generally, people who buy off-the-shelf software don’t want to be part of a Grand Development Experiment; they want something that anticipates their needs. As a customer, the only thing better than getting feature requests done quickly is getting them instantaneously because they’re already in the product, because it was designed thoughtfully and extensively usability- and beta-tested before being inflicted on the world. If you have (or want) a large number of paying customers, prefer less frequent releases.

If you ship an anemic commercial program just to get something out the door so you can “start listening to customers,” what you’ll hear those customers saying is “it doesn’t do very much,” which you might think is OK. Hey, it’s 1.0. But then if you release 2.0 four months later, everybody’s going to think, “That feeble program? What am I, supposed to keep evaluating it every four months just to see if it’s gotten better yet?!” And in fact five years down the line, people will still remember their first impression of 1.0, and it will be almost impossible to get them to reevaluate. Think about what happened to poor Marimba. They launched their company with infinite VC in the days of hyper-Java-hype, having lured the key developers from the Java team at Sun. They had a CEO, Kim Polese, who was brilliant at public relations; when she was marketing Java she had Danny Hillis making speeches about how Java was the next step in human evolution; George Gilder wrote these breathless articles about how Java was going to completely upturn the very nature of human civilization. Compared to Java, we were to believe, monotheism, for example, was just a wee blip. Polese is that good. So when Marimba Castanet launched it probably had more unearned hype than any product in history, but the developers had only been working on it for a total of … four months. We all downloaded it and discovered that — tada! — it was a list box that downloaded software. (What do you expect from four months of development?) Big whoop. The disappointment was so thick you could cut it with butter. And here it is, six years later, ask anybody what Castanet is and they’ll tell you it’s a list box that downloads software. Hardly anyone bothered to reevaluate it, and Marimba has had six years to write code; I’m sure it’s just the coolest thing now but, honestly, who has time to find out? Let me tell you a little secret: our strategy for CityDesk is to avoid massive PR until 2.0 is out. That’s the version that we want everybody on earth to get their first impressions from. In the meantime we’ll do quiet guerilla marketing, and anybody who finds it will discover that it’s a completely spiffy program that solves a lot of problems, but Arnold Toynbee won’t have to rewrite anything.

[Image]With most commercial software, you’ll discover that the process of designing, prototyping, integrating, fixing bugs, running a full alpha and beta cycles, creating documentation, and so forth takes 6-9 months. In fact if you try to do a full release every year, you only have time for about 3 months worth of new code development. Software that is upgraded annually usually doesn’t feel like it has enough new features to justify the upgrade. (Corel PhotoPaint and Intuit Quickbooks are particularly egregious examples of this; they have a new “major” version every year which is rarely worth buying). As a result many people have learned by now to skip every other release. You don’t want your customers getting in that habit. If you stretch out your schedule to 15 or 18 months between releases, you get 6 months of new features instead of 3 months worth, which makes your upgrade a lot more compelling.

OK, if 15 months is so good, wouldn’t 24 months be better? Maybe. Some companies can get away with it, if they are major leaders in their category. PhotoShop seems to get away with it. But as soon as an application starts to feel old, people stop buying it because they are expecting that new version Any Day Now. This can cause serious cash flow problems for a software business. And of course you may have competitors nipping at your heels.

For large platform software — operating systems, compilers, web browsers, DBMSs — the hardest part of the development process is maintaining compatibility with thousands or millions of existing applications or hardware. When a new version of Windows comes out, you very rarely hear about backwards compatibility problems. The only way they can achieve this is with insane amounts of testing that make the construction of the Panama Canal seem like a weekend do-it-yourself project. Given the typical 3 year cycle between major Windows releases, almost all of that time is spent in the boring integration and testing phase, not writing new features. Releasing new versions any more often than that is just not realistic. And it would drive people crazy. Third party software and hardware developers would simply revolt if they had to test against lots of little incremental releases of an operating system. For Systems With Millions of Customers and Millions of Integration Points, Prefer Rare Releases. You can do it like Apache: one release at the beginning of the Internet Bubble, and one release at the end. Perfect.

Souk - Damascus GateIf you have a lot of validation and unit tests, and if you write your software carefully, you may get to the point where any daily build is almost high enough quality to ship. This is certainly something to strive for. Even if you’re planning the next release for three years from now, the competitive landscape may suddenly change and there may be a good reason to do a quick interim release to react to a competitor. When your wife is about to give birth, it’s not really a good idea to take apart your car’s engine. Instead build a new one on the side and don’t hook it up until it’s perfect.

But don’t overestimate what you can accomplish by keeping high quality daily builds. Even if you are permanently at zero bugs, if your software has to run In The Wild, you’re never going to find all the compatibility bugs and Windows 95 bugs and It-Doesn’t-Work-With-Large-Fonts-Turned-On bugs until you do a few betas, and there is no realistic way to do a beta cycle in less than 8 weeks.

One final thought. If your software is delivered as a service over the web, like ebay or PayPal, theoretically there’s nothing to stop you from frequent small releases, but this might not be the best thing to do. Remember the cardinal rule of usability: an application is usable if it behaves the way that the user expected it to behave. If you keep changing things around every week, it won’t be so predictable, so it won’t be so usable. (And don’t think you can get around this by having one of those annoying screens with a paragraph of text saying Warning! The UI has changed! Nobody reads those.) From a usability perspective a better approach would probably be less frequent releases that include a bunch of changes all at once, where you make an effort to change the visuals of the whole site so that it all looks weird and users intuit that things have changed a lot and they have to be careful.