Evidence Based Scheduling

Software developers don’t really like to make schedules. Usually, they try to get away without one. “It’ll be done when it’s done!” they say, expecting that such a brave, funny zinger will reduce their boss to a fit of giggles, and in the ensuing joviality, the schedule will be forgotten.

Most of the schedules you do see are halfhearted attempts. They’re stored on a file share somewhere and completely forgotten. When these teams ship, two years late, that weird guy with the file cabinet in his office brings the old printout to the post mortem, and everyone has a good laugh. “Hey look! We allowed two weeks for rewriting from scratch in Ruby!”

Hilarious! If you’re still in business.

You want to be spending your time on things that get the most bang for the buck. And you can’t figure out how much buck your bang is going to cost without knowing how long it’s going to take. When you have to decide between the “animated paperclip” feature and the “more financial functions” feature, you really need to know how much time each will take.

Why won’t developers make schedules? Two reasons. One: it’s a pain in the butt. Two: nobody believes the schedule is realistic. Why go to all the trouble of working on a schedule if it’s not going to be right?

Over the last year or so at Fog Creek we’ve been developing a system that’s so easy even our grouchiest developers are willing to go along with it. And as far as we can tell, it produces extremely reliable schedules. It’s called Evidence-Based Scheduling, or EBS. You gather evidence, mostly from historical timesheet data, that you feed back into your schedules. What you get is not just one ship date: you get a confidence distribution curve, showing the probability that you will ship on any given date. It looks like this:

The steeper the curve, the more confident you are that the ship date is real.

Here’s how you do it.

1) Break ‘er down

When I see a schedule measured in days, or even weeks, I know it’s not going to work. You have to break your schedule into very small tasks that can be measured in hours. Nothing longer than 16 hours.

This forces you to actually figure out what you are going to do. Write subroutine foo. Create this dialog box. Parse the Fizzbott file. Individual development tasks are easy to estimate, because you’ve written subroutines, created dialogs, and parsed files before.

If you are sloppy, and pick big three-week tasks (e.g., “Implement Ajax photo editor”), then you haven’t thought about what you are going to do. In detail. Step by step. And when you haven’t thought about what you’re going to do, you can’t know how long it will take.

Setting a 16-hour maximum forces you to design the damn feature. If you have a hand-wavy three week feature called “Ajax photo editor” without a detailed design, I’m sorry to be the one to break it to you but you are officially doomed. You never thought about the steps it’s going to take and you’re sure to be forgetting a lot of them.

2) Track elapsed time

It’s hard to get individual estimates exactly right. How do you account for interruptions, unpredictable bugs, status meetings, and the semiannual Windows Tithe Day when you have to reinstall everything from scratch on your main development box? Heck, even without all that stuff, how can you tell exactly how long it’s going to take to implement a given subroutine?

You can’t, really.

So, keep timesheets. Keep track of how long you spend working on each task. Then you can go back and see how long things took relative to the estimate. For each developer, you’ll be collecting data like this:

Each point on the chart is one completed task, with the estimate and actual times for that task. When you divide estimate by actual, you get velocity: how fast the task was done relative to estimate. Over time, for each developer, you’ll collect a history of velocities.

  • The mythical perfect estimator, who exists only in your imagination, always gets every estimate exactly right. So their velocity history is {1, 1, 1, 1, 1, …}
  • A typical bad estimator has velocities all over the map, for example {0.1, 0.5, 1.7, 0.2, 1.2, 0.9, 13.0}
  • Most estimators get the scale wrong but the relative estimates right. Everything takes longer than expected, because the estimate didn’t account for bug fixing, committee meetings, coffee breaks, and that crazy boss who interrupts all the time. This common estimator has very consistent velocities, but they’re below 1.0. For example, {0.6, 0.5, 0.6, 0.6, 0.5, 0.6, 0.7, 0.6}

As estimators gain more experience, their estimating skills improve. So throw away any velocities older than, say, six months.

If you have a new estimator on your team, who doesn’t have a track record, assume the worst: give them a fake history with a wide range of velocities, until they’ve finished a half-dozen real tasks.

3) Simulate the future

Rather than just adding up estimates to get a single ship date, which sounds right but gives you a profoundly wrong result, you’re going to use the Monte Carlo method to simulate many possible futures. In a Monte Carlo simulation, you can create 100 possible scenarios for the future. Each of these possible futures has 1% probability, so you can make a chart of the probability that you will ship by any given date.

While calculating each possible future for a given developer, you’re going divide each task’s estimate by a randomly-selected velocity from that developer’s historical velocities, which we’ve been gathering in step 2. Here’s one sample future:

Estimate: 4 8 2 8 16
Random Velocity: 0.6 0.5 0.6 0.6 0.5 Total:
E/V: 6.7 16 3.3 13.3 32 71.3

Do that 100 times; each total has 1% probability, and now you can figure out the probability that you will ship on any given date.

Now watch what happens:

  • In the case of the mythical perfect estimator, all velocities are 1. Dividing by a velocity which is always 1 has no effect. Thus, all rounds of the simulation give the same ship date, and that ship date has 100% probability. Just like in the fairy tales!
  • The bad estimator’s velocities are all over the map. 0.1 and 13.0 are just as likely. Each round of the simulation is going to produce a very different result, because when you divide by random velocities you get very different numbers each time. The probability distribution curve you get will be very shallow, showing an equal chance of shipping tomorrow or in the far future. That’s still useful information to get, by the way: it tells you that you shouldn’t have confidence in the predicted ship dates.
  • The common estimator has a lot of velocities that are pretty close to each other, for example, {0.6, 0.5, 0.6, 0.6, 0.5, 0.6, 0.7, 0.6}. When you divide by these velocities you increase the amount of time something takes, so in one iteration, an 8-hour task might 13 hours; in another it might take 15 hours. That compensates for the estimators perpetual optimism. And it compensates precisely, based exactly on this developers actual, proven, historical optimism. And since all the historical velocities are pretty close, hovering around 0.6, when you run each round of the simulation, you’ll get pretty similar numbers, so you’ll wind up with a narrow range of possible ship dates.

In each round of the Monte Carlo simulation, of course, you have to convert the hourly data to calendar data, which means you have to take into account each developer’s work schedule, vacations, holidays, etc. And then you have to see, for each round, which developer is finishing last, because that’s when the whole team will be done. These calculations are painstaking, but luckily, painstaking is what computers are good at.

Obsessive-compulsive disorder not required

What do you do about the boss who interrupts you all the time with long-winded stories about his fishing trips? Or the sales meetings you’re forced to go to even though you have no reason to be there? Coffee breaks? Spending half a day helping the new guy get his dev environment set up?

When Brett and I were developing this technique at Fog Creek, we worried a lot about things that take real time but can’t be predicted in advance. Sometimes, this all adds up to more time than writing code. Should you have estimates for this stuff too, and track it on a time sheet?

Well, yeah, you can, if you want. And Evidence Based Scheduling will work.

But you don’t have to.

It turns out that EBS works so well that all you have to do is keep the clock running on whatever task you were doing when the interruption occurred. As disconcerting as this may sound, EBS produces the best results when you do this.

Let me walk you through a quick example. To make this example as simple as possible, I’m going to imagine a very predictable programmer, John, whose whole job is writing those one-line getter and setter functions that inferior programming languages require. All day long this is all he does:

private int width;
public int getWidth () { return width; }
public void setWidth (int _width} { width = _width; }

I know, I know… it’s a deliberately dumb example, but you know you’ve met someone like this.

Anyway. Each getter or setter takes him 2 hours. So his task estimates look like this:

{2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, … }

Now, this poor guy has a boss who interrupts him every once in a while with a two-hour conversation about marlin fishing. Now, of course, John could have a task on his schedule called “Painful conversations about marlin,” and put that on his timesheet, but this might not be politically prudent. Instead, John just keeps the clock running. So his actual times look like this:

{2, 2, 2, 2, 4, 2, 2, 2, 2, 4, 2, … }

And his velocities are:

{1, 1, 1, 1, 0.5, 1, 1, 1, 1, 0.5, 1, … }

Now think about what happens. In the Monte Carlo simulation, the probability that each estimate will be divided by 0.5 is exactly the same as the probability that John’s boss would interrupt him during any given feature. So EBS produces a correct schedule!

In fact, EBS is far more likely to have accurate evidence about these interruptions than even the most timesheet-obsessive developer. Which is exactly why it works so well. Here’s how I explain this to people. When developers get interrupted, they can either

  1. make a big stink about putting the interruption on their timesheet and in their estimates, so management can see just how much time is being wasted on fishing conversation, or
  2. make a big stink about refusing to put it on their timesheet, just letting the feature they were working on slip, because they refuse to pad their estimates which were perfectly correct with stupid conversation about fishing expeditions to which they weren’t even invited,

… and in either case, EBS gives the same, exactly correct results, no matter which type of passive-aggressive developer you have.

4) Manage your projects actively

Once you’ve got this set up, you can actively manage projects to ship on time. For example, if you sort features out into different priorities, it’s easy to see how much it would help the schedule if you could cut the lower priority features.

You can also look at the distribution of possible ship dates for each developer:

Some developers (like Milton in this picture) may be causing problems because their ship dates are so uncertain: they need to work on learning to estimate better. Other developers (like Jane) have very precise ship dates that are just too late: they need to have some of their work taken off their plate. Other developers (me! yay!) are not on the critical path at all, and can be left in peace.

Scope creep

Assuming you had everything planned down to the last detail when you started work, EBS works great. To be honest, though, you may do some features that you hadn’t planned. You get new ideas, your salespeople sell features you don’t have, and somebody on the board of directors comes up with a cool new idea to make your golf cart GPS application monitor EKGs while golfers are buzzing around the golf course. All this leads to delays that could not have been predicted when you did the original schedule.

Ideally, you have a bunch of buffer for this. In fact, go ahead and build buffer into your original schedule for:

  1. New feature ideas
  2. Responding to the competition
  3. Integration (getting everyone’s code to work together when it’s merged)
  4. Debugging time
  5. Usability testing (and incorporating the results of those tests into the product).
  6. Beta tests

So now, when new features come up, you can slice off a piece of the appropriate buffer and use it for the new feature.

What happens if you’re still adding features and you’ve run out of buffer? Well, now the ship dates you get out of EBS start slipping. You should take a snapshot of the ship date confidence distribution every night, so that you can track this over time:

The x-axis is when the calculation was done; the y-axis is the ship date. There are three curves here: the top one is the 95% probability date, the middle is 50% and the bottom is 5%. So, the closer the curves are to one another, the narrower the range of possible ship dates.

If you see ship date getting later and later (rising curves), you’re in trouble. If it’s getting later by more than one day per day, you’re adding work faster than you’re completing work, and you’ll never be done. You can also look and see if the ship date confidence distribution is getting tighter (the curves are converging), which it should be if you’re really converging on a date.

While we’re at it

Here are a few more things I’ve learned over the years about schedules.

1) Only the programmer doing the work can create the estimate. Any system where management writes a schedule and hands it off to programmers is doomed to fail. Only the programmer who is going to implement a feature can figure out what steps they will need to take to implement that feature.

2) Fix bugs as you find them, and charge the time back to the original task. You can’t schedule a single bug fix in advance, because you don’t know what bugs you’re going to have. When bugs are found in new code, charge the time to the original task that you implemented incorrectly. This will help EBS predict the time it takes to get fully debugged code, not just working code.

3) Don’t let managers badger developers into shorter estimates. Many rookie software managers think that they can “motivate” their programmers to work faster by giving them nice, “tight” (unrealistically short) schedules. I think this kind of motivation is brain-dead. When I’m behind schedule, I feel doomed and depressed and unmotivated. When I’m working ahead of schedule, I’m cheerful and productive. The schedule is not the place to play psychological games.

Why do managers try this?

When the project begins, the technical managers go off, meet with the business people, and come up with a list of features they think would take about three months, but which would really take twelve. When you think of writing code without thinking about all the steps you have to take, it always seems like it will take n time, when in reality it will probably take more like 4n time. When you do a real schedule, you add up all the tasks and realize that the project is going to take much longer than originally thought. The business people are unhappy.

Inept managers try to address this by figuring out how to get people to work faster. This is not very realistic. You might be able to hire more people, but they need to get up to speed and will probably be working at 50% efficiency for several months (and dragging down the efficiency of the people who have to mentor them).

You might be able to get 10% more raw code out of people temporarily at the cost of having them burn out 100% in a year. Not a big gain, and it’s a bit like eating your seed corn. Of course, when you overwork people, debugging time doubles and a late project becomes later. Splendid karma.

But you can never get 4n from n, ever, and if you think you can, please email me the stock symbol for your company so I can short it.

4) A schedule is a box of wood blocks. If you have a bunch of wood blocks, and you can’t fit them into a box, you have two choices: get a bigger box, or remove some blocks. If you wanted to ship in six months, but you have twelve months on the schedule, you are either going to have to delay shipping, or find some features to delete. You just can’t shrink the blocks, and if you pretend you can, then you are merely depriving yourself of a useful opportunity to actually see into the future by lying to yourself about what you see there.

Now that I mention it, one of the great benefits of realistic schedules is that you are forced to delete features. Why is this good?

Suppose you have two features in mind. One is really useful and will make your product really great. The other is really easy and the programmers can’t wait to code it up (”Look! <blink>!”), but it serves no useful purpose.

If you don’t make a schedule, the programmers will do the easy/fun feature first. Then they’ll run out of time, and you will have no choice but to slip the schedule to do the useful/important feature.

If you do make a schedule, even before you start working, you’ll realize that you have to cut something, so you’ll cut the easy/fun feature and just do the useful/important feature. By forcing yourself to chose some features to cut, you wind up making a more powerful, better product with a better mix of good features that ships sooner.

Way back when I was working on Excel 5, our initial feature list was huge and would have gone way over schedule. “Oh my!” we thought. “Those are all super important features! How can we live without a macro editing wizard?”

As it turns out, we had no choice, and we cut what we thought was “to the bone” to make the schedule. Everybody felt unhappy about the cuts. To make people feel better, we told ourselves that we weren’t cutting the features, we were simply deferring them to Excel 6.

As Excel 5 was nearing completion, I started working on the Excel 6 spec with a colleague, Eric Michelman. We sat down to go through the list of “Excel 6” features that had been punted from the Excel 5 schedule. Guess what? It was the shoddiest list of features you could imagine. Not one of those features was worth doing. I don’t think a single one of them ever was. The process of culling features to fit a schedule was the best thing we could have done. If we hadn’t done this, Excel 5 would have taken twice as long and included 50% useless crap features that would have had to be supported, for backwards compatibility, until the end of time.

Summary

Using Evidence-Based Scheduling is pretty easy: it will take you a day or two at the beginning of every iteration to produce detailed estimates, and it’ll take a few seconds every day to record when you start working on a new task on a timesheet. The benefits, though, are huge: realistic schedules.

Realistic schedules are the key to creating good software. It forces you to do the best features first and allows you to make the right decisions about what to build. Which makes your product better, your boss happier, delights your customers, and—best of all—lets you go home at five o’clock.

P.S.

Evidence Based Scheduling is built into FogBugz 6.0.

The Identity Management Method

When you’re trying to get a team all working in the same direction, we’ve seen that Command and Control management and Econ 101 management both fail pretty badly in high tech, knowledge- oriented teams.

That leaves a technique that I’m going to have to call The Identity Method. The goal here is to manage by making people identify with the goals you’re trying to achieve. That’s a lot trickier than the other methods, and it requires some serious interpersonal skills to pull off. But if you do it right, it works better than any other method.

The problem with Econ 101 management is that it subverts intrinsic motivation. The Identity Method is a way to create intrinsic motivation.

To be an Identity Method manager, you have to summon all the social skills you have to make your employees identify with the goals of the organization, so that they are highly motivated, then you need to give them the information they need to steer in the right direction.

How do you make people identify with the organization?

It helps if the organizational goals are virtuous, or perceived as virtuous, in some way. Apple creates almost fanatic identification, almost entirely through a narrative that started with a single Superbowl ad in 1984: we are against totalitarianism. Doesn’t seem like a particularly bold position to take, but it worked. Here at Fog Creek, we stand bravely in opposition to killing kittens. Yaaaay!

A method I’m pretty comfortable with is eating together. I’ve always made a point of eating lunch with my coworkers, and at Fog Creek we serve catered lunches for the whole team every day and eat together at one big table. It’s hard to understate what a big impact this has on making the company feel like a family, in the good way, I think. In six years, nobody has ever quit.

I’m probably going to freak out some of our summer interns by admitting this, but one the goals of our internship program is to make people identify as New Yorkers, so they’re more comfortable with the idea of moving here after college and working for us full-time. We do this through a pretty exhausting list of extra-curricular summer activities: two Broadway shows, a trip to the Top of the Rock, a boat ride around Manhattan, a Yankees game, an open house so they can meet more New Yorkers, and a trip to a museum; Michael and I host parties in our apartments, both as a way of welcoming the interns but also as a way for interns to visualize living in an apartment in New York, not just the dorm we stuck them in.

In general, Identity Management requires you to create a cohesive, jelled team that feels like a family, so that people have a sense of loyalty and commitment to their coworkers.

The second part, though, is to give people the information they need to steer the organization in the right direction.

Earlier today Brett came into my office to discuss ship dates for FogBugz 6.0. He was sort of leaning towards April 2007; I was sort of leaning towards December 2006. Of course, if we shipped in April, we would have time to do a lot more polishing, and improve a lot of areas of the product; if we shipped in December, we’d probably have to cut a bunch of nice new features.

What I explained to Brett, though, is that we want to hire six new people in the spring, and the chances that we’ll be able to afford them without FogBugz 6.0 are much smaller. So the way I concluded the meeting with Brett was to make him understand the exact financial motivations I have for shipping earlier, and now that he knows that, I’m confident he’ll make the right decision… not necessarily my decision. Maybe we’ll have a big upswing in sales without FogBugz 6.0, and now that Brett understands the basic financial parameters, he’ll realize that maybe that means we can hold 6.0 for a few more features. The point being that by sharing information, I can get Brett to do the right thing for Fog Creek even if circumstances change. If I tried to push him around by offering him a cash reward for every day before April that he ships, his incentive would be to dump the existing buggy development build on the public tonight. If I tried to push him around using Command and Control management by ordering him to ship bug free code on time, dammit, he might do it, but he’d hate his job and leave.

Conclusion

There are as many different styles of management as there are managers. I’ve identified three major styles: two easy, dysfunctional styles and one hard, functional style, but the truth is that many development shops manage in more of an ad-hoc, “whatever works” way that may change from day to day or person to person.

The Econ 101 Management Method

Joke: A poor Jew lived in the shtetl in 19th century Russia. A Cossack comes up to him on horseback.

“What are you feeding that chicken?” asks the Cossack.

“Just some bread crumbs,” replies the Jew.

“How dare you feed a fine Russian chicken such lowly food!” says the Cossack, and hits the Jew with a stick.

The next day the Cossack comes back. “Now what are you feeding that chicken?” ask the Jew.

“Well, I give him three courses. There’s freshly cut grass, fine sturgeon caviar, and a small bowl of heavy cream sprinkled with imported French chocolate truffles for dessert.”

“Idiot!” says the Cossack, beating the Jew with a stick. “How dare you waste good food on a lowly chicken!”

On the third day, the Cossack again asks, “What are you feeding that chicken?”

“Nothing!” pleads the Jew. “I give him a kopeck and he buys whatever he wants.”

(pause for laughter)

(no?)

(ba dum dum)

(still no laughter)

(oh well).

I use the term “Econ 101” a little bit tongue-in-cheek. For my non-American readers: most US college departments have a course numbered “101” which is the basic introductory course for any field. Econ 101 management is the style used by people who know just enough economic theory to be dangerous.

The Econ 101 manager assumes that everyone is motivated by money, and that the best way to get people to do what you want them to do is to give them financial rewards and punishments to create incentives.

For example, AOL might pay their call-center people for every customer they persuade not to cancel their subscription.

A software company might give bonuses to programmers who create the fewest bugs.

It works about as well as giving your chickens money to buy their own food.

One big problem is that it replaces intrinsic motivation with extrinsic motivation.

Intrinsic motivation is your own, natural desire to do things well. People usually start out with a lot of intrinsic motivation. They want to do a good job. They want to help people understand that it’s in their best interest to keep paying AOL $24 a month. They want to write less-buggy code.

Extrinsic motivation is a motivation that comes from outside, like when you’re paid to achieve something specific.

Intrinsic motivation is much stronger than extrinsic motivation. People work much harder at things that they actually want to do.  That’s not very controversial.

But when you offer people money to do things that they wanted to do, anyway, they suffer from something called the Overjustification Effect. “I must be writing bug-free code because I like the money I get for it,” they think, and the extrinsic motivation displaces the intrinsic motivation. Since extrinsic motivation is a much weaker effect, the net result is that you’ve actually reduced their desire to do a good job. When you stop paying the bonus, or when they decide they don’t care that much about the money, they no longer think that they care about bug free code.

Another big problem with Econ 101 management is the tendency for people to find local maxima. They’ll find some way to optimize for the specific thing you’re paying them, without actually achieving the thing you really want.

So for example your customer retention specialist, in his desire to earn the bonus associated with maintaining a customer, will drive the customer so crazy that the New York Times will run a big front page story about how nasty your customer “service” is. Although his behavior maximizes the thing you’re paying him for (customer retention) it doesn’t maximize the thing you really care about (profit). And then you try to reward him for the company profit, say, by giving him 13 shares of stock, and you realize that it’s not really something he controls, so it’s a waste of time.

When you use Econ 101 management, you’re encouraging developers to game the system.

Suppose you decide to pay a bonus to the developer with the fewest bugs. Now every time a tester tries to report a bug, it becomes a big argument, and usually the developer convinces the tester that it’s not really a bug. Or the tester agrees to report the bug “informally” to the developer before writing it up in the bug tracking system. And now nobody uses the bug tracking system. The bug count goes way down, but the number of bugs stays the same.

Developers are clever this way. Whatever you try to measure, they’ll find a way to maximize, and you’ll never quite get what you want.

Robert Austin, in his book Measuring and Managing Performance in Organizations, says there are two phases when you introduce new performance metrics. At first, you actually get what you wanted, because nobody has figured out how to cheat. In the second phase, you actually get something worse, as everyone figures out the trick to maximizing the thing that you’re measuring, even at the cost of ruining the company.

Worse, Econ 101 managers think that they can somehow avoid this situation just by tweaking the metrics. Dr. Austin’s conclusion is that you just can’t. It never works. No matter how much you try to adjust the metrics to reflect what you think you want, it always backfires.

The biggest problem with Econ 101 management, though, is that it’s not management at all: it’s really more of an abdication of management. A deliberate refusal to figure out how things can be made better. It’s a sign that management simply doesn’t know how to teach people to do better work, so they force everybody in the system to come up with their own way of doing it.

Instead of training developers on techniques of writing reliable code, you just absolve yourself of responsibility by paying them if they do. Now every developer has to figure it out on their own.

For more mundane tasks, working the counter at Starbucks or answering phone calls at AOL, it’s pretty unlikely that the average worker will figure out a better way of doing things on their own. You can go into any coffee shop in the country and order a short soy caramel latte extra-hot, and you’ll find that you have to keep repeating your order again and again: once to the coffee maker, again to the coffee maker when they forgot what you said, and finally to the cashier so they can figure out what to charge you. That’s the result of nobody telling the workers a better way. Nobody figures it out, except Starbucks, where the standard training involves a complete system of naming, writing things on cups, and calling out orders which insures that customers only have to specify their drink orders once. The system, invented by Starbucks HQ, works great, but workers at the other chains never, ever come up with it on their own.

Your customer service people spend most of the day talking to customers. They don’t have the time, the inclination, or the training to figure out better ways to do things. Nobody in the customer retention crew is going to be able to keep statistics and measure which customer retention techniques work best while pissing off the fewest bloggers. They just don’t care enough, they’re not smart enough, they don’t have enough information, and they are too busy with their real job.

As a manager it’s your job to figure out a system. That’s Why You Get The Big Bucks.

If you read a little bit too much Ayn Rand as a kid, or if you took one semester of Economics, before they explained that utility is not measured in dollars, you may think that setting up simplified bonus schemes and Pay For Performance is a pretty neat way to manage. But it doesn’t work. Start doing your job managing and stop feeding your chickens kopecks.

“Joel!” you yell. “Yesterday you told us that the developers should make all the decisions. Today you’re telling us that the managers should make all the decisions. What’s up with that?”

Mmm, not exactly. Yesterday I told you that your developers, the leaves in the tree, have the most information; micromanagement or Command and Control barking out orders is likely to cause non-optimal results. Today I’m telling you that when you’re creating a system, you can’t abdicate your responsibility to train your people by bribing them. Management, in general, needs to set up the system so that people can get things done, it needs to avoid displacing intrinsic motivation with extrinsic motivation, and it won’t get very far using fear and barking out specific orders.

Now that I’ve shot down Command and Control management and Econ 101 management, there’s one more method managers can use to get people moving in the right direction. I call it the Identity method and I’ll talk about it more tomorrow.

The Command and Control Management Method

Frederick the Great [PDF]: “Soldiers should fear their officers more than all the dangers to which they are exposed…. Good will can never induce the common soldier to stand up to such dangers; he will only do so through fear.”

The Command and Control form of management is based on military management. Primarily, the idea is that people do what you tell them to do, and if they don’t, you yell at them until they do, and if they still don’t, you throw them in the brig for a while, and if that doesn’t teach them, you put them in charge of peeling onions on a submarine, sharing two cubit feet of personal space with a lad from a farm who really never quite learned about brushing his teeth.

There are a million great techniques you can use. Rent the movies Biloxi Blues and An Officer and a Gentleman for some ideas.

Some managers use this technique because they actually learned it in the military. Others grew up in authoritarian households or countries and think it’s a natural way to gain compliance. Others just don’t know any better. Hey, it works for the military, it should work for an internet startup!

There are, it turns out, three drawbacks with this method in a high tech team.

First of all, people don’t really like it very much, least of all smarty-pants software developers, who are, actually, pretty smart and are used to thinking they know more than everyone else, for perfectly good reasons, because it happens to be true, and so it really, really bothers them when they’re commanded to do something “because.” But that’s not really a good enough reason to discard this method… we’re trying to be rational here. High tech teams have many goals but making everyone happy is rarely goal number one.

A more practical drawback with Command and Control is that management literally does not have enough time to micromanage at this level, because there simply aren’t enough managers. In the military, it’s possible to give an order simultaneously to a large team of people because it’s common that everyone is doing the same thing. “Clean your guns!” you can say, to a squad of 28, and then go take a brief nap and have a cool iced tea on the Officer’s Club veranda. In software development teams everybody is working on something else, so attempts to micromanage turn into hit and run micromanagement. That’s where you micromanage one developer in a spurt of activity and then suddenly disappear from that developer’s life for a couple of weeks while you run around micromanaging other developers. The problem with hit and run micromanagement is that you don’t stick around long enough to see why your decisions are not working or to correct course. Effectively, all you accomplish is to knock your poor programmers off the train track every once in a while, so they spend the next week finding all their train cars and putting them back on the tracks and lining everything up again, a little bit battered from the experience.

The third drawback is that in a high tech company the individual contributors always have more information than the “leaders,” so they are really in the best position to make decisions. When the boss wanders into an office where two developers have been arguing for two hours about the best way to compress an image, the person with the least information is the boss, so that’s the last person you’d want making a technical decision. I remember when Mike Maples was my great grand-boss, in charge of Microsoft Applications, he was adamant about refusing to take sides on technical issues. Eventually people learned that they shouldn’t come to him to adjudicate. This forced people to debate the issue on the merits and issues were always resolved in favor of the person who was better at arguing, er, I mean, issues were always resolved in the best possible way.

If Command and Control is such a bad way to run a team, why does the military use it?

This was explained to me in NCO school. I was in the Israeli paratroopers in 1986. Probably the worst paratrooper they ever had, now that I think back.

There are several standing orders for soldiers. Number one: if you are in a mine field, freeze. Makes sense, right? It was drilled into you repeatedly during basic training. Every once in a while the instructor would shout out “Mine!” and everybody had to freeze just so you would get in the habit.

Standing order number two: when attacked, run towards your attackers while shooting. The shooting makes them take cover so they can’t fire at you. Running towards them causes you to get closer to them, which makes it easier to aim at them, which makes it easier to kill them. This standing order makes a lot of sense, too.

OK, now for the Interview Question. What do you do if you’re in a minefield, and people start shooting at you?

This is not such a hypothetical situation; it’s a really annoying way to get caught in an ambush.

The correct answer, it turns out, is that you ignore the minefield, and run towards the attackers while shooting.

The rationale behind this is that if you freeze, they’ll pick you off one at a time until you’re all dead, but if you charge, only some of you will die by running over mines, so for the greater good, that’s what you have to do.

The trouble is that no rational soldier would charge under such circumstances. Each individual soldier has an enormous incentive to cheat: freeze in place and let the other, more macho soldiers do the charging. It’s sort of like a Prisoners’ Dilemma.

In life or death situations, the military needs to make sure that they can shout orders and soldiers will obey them even if the orders are suicidal. That means soldiers need to be programmed to be obedient in a way which is not really all that important for, say, a software company.

In other words, the military uses Command and Control because it’s the only way to get 18 year olds to charge through a minefield, not because they think it’s the best management method for every situation.

In particular, in software development teams where good developers can work anywhere they want, playing soldier is going to get pretty tedious and you’re not really going to keep anyone on your team.

Three Management Methods (Introduction)

If you want to lead a team, a company, an army, or a country, the primary problem you face is getting everyone moving in the same direction, which is really just a polite way of saying “getting people to do what you want.”

Think of it this way. As soon as your team consists of more than one person, you’re going to have different people with different agendas. They want different things than you want. If you’re a startup founder, you might want to make a lot of money quickly so you can retire early and spend the next couple of decades going to conferences for women bloggers. So you might spend most of your time driving around Sand Hill Road talking to VCs who might buy the company and flip it to Yahoo!. But Janice the Programmer, one of your employees, doesn’t care about selling out to Yahoo!, because she’s not going to make any money that way. What she cares about is writing code in the latest coolest new programming language, because it’s fun to learn a new thing. Meanwhile your CFO is entirely driven by the need to get out of the same cubicle he has been sharing with the system administrator, Trekkie Monster, and so he’s working up a new budget proposal that shows just how much money you would save by moving to larger office space that’s two minutes from his house, what a coincidence!

The problem of getting people to move in your direction (or, at least, the same direction) is not unique to startups, of course. It’s the same fundamental problem that a political leader faces when they get elected after promising to eliminate waste, corruption, and fraud in government. The mayor wants to make sure that it’s easy to get city approval of a new building project. The city building inspectors want to keep getting the bribes they have grown accustomed to.

And it’s the same problem that a military leader faces. They might want a team of soldiers to charge at the enemy, even when every individual soldier would really just rather cower behind a rock and let the others do the charging.

Here are three common approaches you might take:

  • The Command and Control Method
  • The Econ 101 Method
  • The Identity Method

You will certainly find other methods of management in the wild (there’s the exotic “Devil Wears Prada” Method, the Jihad Method, the Charismatic Cult Method, and the Lurch From One Method To Another Method) but over the next three days, I’m going to examine these three popular methods and explore their pros and cons.

The Development Abstraction Layer

A young man comes to town. He is reasonably good looking, has a little money in his pocket. He finds it easy to talk to women.

He doesn’t speak much about his past, but it is clear that he spent a lot of time in a soulless big company.

He is naturally friendly and outgoing, and quietly confident without being arrogant. So he finds it easy to pick up small gigs from the job board at the local Programmer’s Cafe. But he rapidly loses interest in insurance database projects, vanity web pages for housewives, and financial calculation engines.

After a year, he calculates that he has saved up enough money to pay his modest expenses for a year. So, after consulting with his faithful Alsatian, he sets up a computer in a sunfilled room in his rented apartment above the grocery store and installs a carefully-chosen selection of tools.

One by one, he calls his friends and warns them that if he seems remote over the next months, it is only because he is hard at work.

And he sits down to spin code.

And what code it is. Flawless, artistic, elegant, bug free. The user interface so perfectly mimics a users’ thought process that the people he shows it to at the Programmer’s Cafe hardly notice that there is a user interface. It’s a brilliant piece of work.

Encouraged by the feedback of his peers, he sets up in business and prepares to take orders.

His modesty precludes any pretensions, but after a month, the situation in his bank account is not looking encouraging. So far only three orders have been taken: one from his mother, one from an anonymous benefactor at the Programmer’s Cafe, and the one he submitted himself to test the commerce system.

In the second month, no more orders come in.

This surprises him and leaves him feeling melancholy. At the big company, new products were created on a regular basis, and even if they were inelegant and homely, they still sold in reasonable quantities. One product he worked on there went on to be a big hit.

After a few more months pass, his financial situation starts to look a little bit precarious. His dog looks at him sadly, not quite certain what is wrong, but aware that his face is looking a little bit gaunter than usual, and he seems to be unable to get up the energy to go out with friends, or go shopping to restock the dangerously low larder, or even to bathe.

One Tuesday morning, the local grocer has refused to extend him any more credit, and his banker has long since refused to return his calls.

The big company is not vindictive. They recognize talent, and are happy to hire him back, at a higher salary. Soon he is looking better, he has some new clothes, and he’s got his old confidence back. But something, somewhere, is missing. A spark in his eye. The hope that he might become the master of his own destiny is gone.

Why did he fail? He’s pretty sure he knows. “Marketing,” he says. Like many young technicians, he is apt to say things like, “Microsoft has worse products but better marketing.”

When uttered by a software developer, the term “marketing” simply stands in for all that business stuff: everything they don’t actually understand about creating software and selling it.

This, actually, is not really what “marketing” means. Actually Microsoft has pretty terrible marketing. Can you imagine those dinosaur ads actually making someone want to buy Microsoft Office?

Software is a conversation, between the software developer and the user. But for that conversation to happen requires a lot of work beyond the software development. It takes marketing, yes, but also sales, and public relations, and an office, and a network, and infrastructure, and air conditioning in the office, and customer service, and accounting, and a bunch of other support tasks.

But what do software developers do? They design and write code, they layout screens, they debug, they integrate, and they check things into the source code control repository.

The level a programmer works at (say, Emacs) is too abstract to support a business. Developers working at the developer abstraction layer need an implementation layer — an organization that takes their code and turns it into products. Dolly Parton, working at the “singing a nice song” layer, needs a huge implementation layer too, to make the records and book the concert halls and take the tickets and set up the audio gear and promote the records and collect the royalties.

Any successful software company is going to consist of a thin layer of developers, creating software, spread across the top of a big abstract administrative organization.

The abstraction exists solely to create the illusion that the daily activities of a programmer (design and writing code, checking in code, debugging, etc.) are all that it takes to create software products and bring them to market. Which gets me to the most important point of this essay:

Your first priority as the manager of a software team is building the development abstraction layer.

Most new software managers miss this point. They keep thinking of the traditional, Command-and-Conquer model of management that they learned from Hollywood movies.

According to Command-and-Conquer, managers-slash-leaders figure out where the business is going to go, and then issue the appropriate orders to their lieutenants to move the business in that direction. Their lieutenants in turn divide up the tasks into smaller chunks and command their reports to implement them. This continues down the org-chart until eventually someone at the bottom actually does some work. In this model, a programmer is a cog in the machine: a typist who carries out one part of management’s orders.

Some businesses actually run this way. You can always tell when you are dealing with such a business, because the person you are talking to is doing something infuriating and senseless, and they know it, and they might even care, but there’s nothing they can do about it. It’s the airline that loses a million mile customer forever because they refuse to change his non-refundable ticket so he can fly home for a family emergency. It’s the ISP whose service is down more often than it’s up, and when you cancel your account, they keep billing you, and billing you, and billing you, but when you call to complain, you have to call a toll number and wait on hold for an hour, and then they still refuse to refund you, until you start a blog about how badly they suck. It’s the Detroit automaker that long since forgot how to design cars that people might want to buy and instead lurches from marketing strategy to marketing strategy, as if the only reason we don’t buy their crappy cars is because the rebate wasn’t big enough.

Enough.

Forget it. The command-hierarchy system of management has been tried, and it seemed to work for a while in the 1920s, competing against peddlers pushing carts, but it’s not good enough for the 21st century. For software companies, you need to use a different model.

With a software company, the first priority of management needs to be creating that abstraction for the programmers.

If a programmer somewhere is worrying about a broken chair, or waiting on hold with Dell to order a new computer, the abstraction has sprung a leak.

Think of your development abstraction layer as a big, beautiful yacht with insanely powerful motors. It’s impeccably maintained. Gourmet meals are served like clockwork. The staterooms have twice-daily maid service. The navigation maps are always up to date. The GPS and the radar always work and if they break there’s a spare below deck. Standing on the bridge, you have programmers who really only think about speed, direction, and whether to have Tuna or Salmon for lunch. Meanwhile a large team of professionals in starched white uniforms tiptoes around quietly below deck, keeping everything running, filling the gas tanks, scraping off barnacles, ironing the napkins for lunch. The support staff knows what to do but they take their cues from a salty old fart who nods ever so slightly in certain directions to coordinate the whole symphony so that the programmers can abstract away everything about the yacht except speed, direction, and what they want for lunch.

Management, in a software company, is primarily responsible for creating abstractions for programmers. We build the yacht, we service the yacht, we are the yacht, but we don’t steer the yacht. Everything we do comes down to providing a non-leaky abstraction for the programmers so that they can create great code and that code can get into the hands of customers who benefit from it.

Programmers need a Subversion repository. Getting a Subversion repository means you need a network, and a server, which has to be bought, installed, backed up, and provisioned with uninterruptible power, and that server generates a lot of heat, which means it need to be in a room with an extra air conditioner, and that air conditioner needs access to the outside of the building, which means installing an 80 pound fan unit on the wall outside the building, which makes the building owners nervous, so they need to bring their engineer around, to negotiate where the air conditioner unit will go (decision: on the outside wall, up here on the 18th floor, at the most inconvenient place possible), and the building gets their lawyers involved, because we’re going to have to sign away our firstborn to be allowed to do this, and then the air conditioning installer guys show up with rigging gear that wouldn’t be out of place in a Barbie play-set, which makes our construction foreman nervous, and he doesn’t allow them to climb out of the 18th floor window in a Mattel harness made out of 1/2″ pink plastic, I swear to God it could be Disco Barbie’s belt, and somebody has to call the building agent again and see why the hell they suddenly realized, 12 weeks into a construction project, that another contract amendment is going to be needed for this goddamned air conditioner that they knew about before Christmas and they only just figured it out, and if your programmers even spend one minute thinking about this that’s one minute too many.

To the software developers on your team, this all needs to be abstracted away as typing svn commit on the command line.

That’s why you have management.

It’s for the kind of stuff that no company can avoid, but if you have your programmers worrying about it, well, management has failed, the same way as a 100 foot yacht has failed if the millionaire owner has to go down into the engine room and, um, build the engine.

You’ve got your typical company started by ex-software salesmen, where everything is Sales Sales Sales and we all exist to drive more sales. These companies can be identified in the wild because they build version 1.0 of the software (somehow) and then completely lose interest in developing new software. Their development team is starved or nonexistent because it never occurred to anyone to build version 2.0… all that management knows how to do is drive more sales.

On the other extreme you have typical software companies built by ex-programmers. These companies are harder to find because in most circumstances they keep quietly to themselves, polishing code in a garret somewhere, which nobody ever finds, and so they fade quietly into oblivion right after the Great Ruby Rewrite, their earth-changing refactoring-code code somehow unappreciated by The People.

Both of these companies can easily be wiped out by a company that’s driven by programmers and organized to put programmers in the driver’s seat, but which have an excellent abstraction that does all the hard work to convert code into products below the decks.

A programmer is most productive with a quiet private office, a great computer, unlimited beverages, an ambient temperature between 68 and 72 degrees (F), no glare on the screen, a chair that’s so comfortable you don’t feel it, an administrator that brings them their mail and orders manuals and books, a system administrator who makes the Internet as available as oxygen, a tester to find the bugs they just can’t see, a graphic designer to make their screens beautiful, a team of marketing people to make the masses want their products, a team of sales people to make sure the masses can get these products, some patient tech support saints who help customers get the product working and help the programmers understand what problems are generating the tech support calls, and about a dozen other support and administrative functions which, in a typical company, add up to about 80% of the payroll. It is not a coincidence that the Roman army had a ratio of four servants for every soldier. This was not decadence. Modern armies probably run 7:1. (Here’s something Pradeep Singh taught me today: if only 20% of your staff is programmers, and you can save 50% on salary by outsourcing programmers to India, well, how much of a competitive advantage are you really going to get out of that 10% savings?)

Management’s primary responsibility to create the illusion that a software company can be run by writing code, because that’s what programmers do. And while it would be great to have programmers who are also great at sales, graphic design, system administration, and cooking, it’s unrealistic. Like teaching a pig to sing, it wastes your time and it annoys the pig.

Microsoft does such a good job at creating this abstraction that Microsoft alumni have a notoriously hard time starting companies. They simply can’t believe how much went on below decks and they have no idea how to reproduce it.

Nobody expects Dolly Parton to know how to plug in a microphone. There’s an incredible infrastructure of managers, musicians, recording technicians, record companies, roadies, hairdressers, and publicists behind her who exist to create the abstraction that when she sings, that’s all it takes for millions of people to hear her song. All the support staff and management that make Dolly Parton possible can do their jobs best by providing the most perfect abstraction: the most perfect illusion that Dolly sings for us. It is her song. When you’re listening to her on your iPod, there’s a huge infrastructure that makes that possible, but the very best thing that infrastructure can do is disappear completely. Provide a leakproof abstraction that Dolly Parton is singing, privately, to us.

Lord Palmerston on Programming

There was a time when if you read one book by Peter Norton, you literally knew everything there was to know about programming the IBM-PC. Over the last 20 years, programmers around the world have been hard at work building abstraction upon abstraction on top of the IBM-PC to make it easier to program and more powerful.

But the law of leaky abstractions means that even as they built the abstractions that are supposed to make programming easier, the sheer amount of stuff you have to know to be a great programmer is expanding all the time.

Becoming proficient, really proficient, in just one programming world takes years. Sure, lots of bright teenagers learn Delphi one week and Python the next week and Perl the next week and think they are proficient. Yet they don’t have the foggiest clue how much they’re missing.

I’ve been working with ASP and VBScript since it first came out. VBScript is the dinkiest language on earth and ASP programming consists of learning about 5 classes, only two of which you use very often. And only now do I finally feel like I know the best way to architect an ASP/VBScript application. I finally think I know where the best place to put database access code is, the best way to use ADO to get recordsets, the best way to separate HTML and code, etc. And I finally use regexps instead of one-off string manipulation functions. Only last week, I learned how to get COM objects out of memory so you can recompile them (without restarting the whole web server).

Fog Creek is too small to have specialists, so when I needed to write a really good installer for FogBUGZ, our ASP/VBScript based product, I drew on several years of C++/MFC experience, and years of experience with Windows APIs, and good Corel PhotoPaint skills to create a neat picture in the corner of the wizard. Then to get FogBUGZ to work perfectly with Unicode, I had to write a little ActiveX control using C++ and ATL, which drew upon years of C++ and COM experience and a week or so learning about character encodings when I implemented that code in CityDesk.

So when we had a weird NT 4.0-only bug, it took me 3 minutes to debug, because I knew how to use VMWare, and I had a clean NT 4.0 machine set up in VMWare, and I knew how to do remote debugging with Visual C++, and I knew to look in the EAX register to get the return value from a function. Someone who was new to this all might have taken an hour or more to debug the same problem, but I already knew a tremendous amount of “stuff” that I’ve been learning, basically, since 1982 when I got my first IBM-PC and that Norton book.

Leaky abstractions mean that we live with a hockey stick learning curve: you can learn 90% of what you use day by day with a week of learning. But the other 10% might take you a couple of years catching up. That’s where the really experienced programmers will shine over the people who say “whatever you want me to do, I can just pick up the book and learn how to do it.” If you’re building a team, it’s OK to have a lot of less experienced programmers cranking out big blocks of code using the abstract tools, but the team is not going to work if you don’t have some really experienced members to do the really hard stuff.

There are a lot of programming worlds, each of which requires a tremendous amount of knowledge for real proficiency. Here are the three I personally know best:

  • MFC/C++/Windows
  • VBScript/ASP
  • Visual Basic

All, basically, what you would call Windows programming. Yes, I’ve written Unix code and Java code, but not very much. My proficiency in Windows programming comes from knowing not just the basic technologies but also the whole supporting infrastructure. So, I claim, I’m really good at Windows programming because I also know COM, ATL, C++, 80×86 Assembler, Windows APIs, IDispatch (OLE Automation), HTML, the DOM, the Internet Explorer object model, Windows NT and Windows 95 internals, LAN Manager and NT networking, including security (ACEs, ACLs, and all that stuff), SQL and SQL Server, Jet and Access, JavaScript, XML, and a few other cheerful facts about the square of the hypotenuse. When I can’t get the StrConv function in VB to do what I want, I bang out an COM control so I can drop into C++ with ATL and call the MLang functions without dropping a beat. It took me years to get to this point.

There are lots of other programming worlds. There’s the world of people developing for BEA Weblogic who know J2EE, Oracle, and all kinds of Java things that I don’t even know enough about to enumerate. There are hard core Macintosh developers who know CodeWarrior, MPW, Toolbox programming in System 6 through X, Cocoa, Carbon, and even nice obsolete things like OpenDoc that don’t help any more.

Very few people, though, know more than one or two worlds, because there’s just so much to learn that unless you have to work in one of these worlds for more than a couple of years, you don’t really grok it all.

But learn you must.

People get kind of miffed when they go on job interviews and get rejected because, for example, they don’t have Win32 (or J2EE, or Mac programming, or whatever) experience. Or they get annoyed because idiot recruiters, who would not know an MSMQ if it bit them in the tailbone, call them up and ask if they “have 5 years MSMQ.”

Until you’ve done Windows programming for a while, you may think that Win32 is just a library, like any other library, you’ll read the book and learn it and call it when you need to. You might think that basic programming, say, your expert C++ skills, are the 90% and all the APIs are the 10% fluff you can catch up on in a few weeks. To these people I humbly suggest: times have changed. The ratio has reversed.

Very few people get to work on low level C algorithms that just move bytes around any more. Most of us spend all our time these days calling APIs, not moving bytes. Someone who is a fantastic C++ coder with no API experience only knows about 10% of what you use every day writing code that runs on an API. When the economy is doing well, this doesn’t matter. You still get jobs, and employers pay the cost of your getting up to speed on the platform. But when the economy is a mess and 600 people apply for every job opening, employers have the luxury of choosing programmers who are already experts at the platform in question. Like programmers who can name four ways to FTP a file from Visual Basic code and the pros and cons of each.

The huge surface area of all these worlds of programming leads to pointless flame wars over whose world is better. Here’s a smug comment somebody anonymously made on my discussion board:

“Just one more reason why I’m glad to be living in the ‘free world.’ Free as in speech (almost) and freedom from pandering to things like setup programs and the registry – just to name a few.”

I think this person was trying to say that in the Linux world they don’t write setup programs. Well, I hate to disappoint you, but you have something just as complicated: imake, make, config files, and all that stuff, and when you’re done, you still distribute applications with a 20KB INSTALL file full of witty instructions like “You’re going to need zlib” (what’s that?) or “This may take a while. Go get some runts.” (Runts are a kind of candy, I think.) And the registry — instead of one big organized hive of name/value pairs, you have a thousand different file formats, one per application, with .whateverrc and foo.conf files living all over the place. And emacs wants you to learn how to program lisp if you’re going to change settings, and each shell wants you to learn its personal dialect of shell script programming if you want to change settings, and on and on.

People who only know one world get really smarmy, and every time they hear about the complications in the other world, it makes them think that their world doesn’t have complications. But they do. You’ve just moved beyond them because you are proficient in them. These worlds are just too big and complicated to compare any more. Lord Palmerston: “The Schleswig-Holstein question is so complicated, only three men in Europe have ever understood it. One was Prince Albert, who is dead. The second was a German professor who became mad. I am the third and I have forgotten all about it.” The software worlds are so huge and complicated and multifaceted that when I see otherwise intelligent people writing blog entries saying something vacuous like “Microsoft is bad at operating systems,” frankly, they just look dumb. Imagine trying to summarize millions of lines of code with hundreds of major feature areas created by thousands of programmers over a decade or two, where no one person can begin to understand even a large portion of it. I’m not even defending Microsoft, I’m just saying that big handwavy generalizations made from a position of deep ignorance is one of the biggest wastes of time on the net today.

Frequent readers, by now, have noticed that I’ve been thinking of the problem of how one might deliver an application on Linux, Macintosh, and Windows without paying disproportionately for the Linux and Macintosh versions. For this you need some kind of cross-platform library.

Java attempted this but Sun didn’t grok GUIs well enough to deliver really slick native-feeling applications. Like the space alien in Star Trek watching Earth through a telescope, they knew exactly what human food was supposed to look like but they didn’t realize it was supposed to taste like something. Java apps have menus in the right places but there are all these keyboard things that don’t work the same way as every other Windows app and their tabbed dialogs look a little scary. And there is no way, no matter how hard you try, to make their menubars look exactly like Excel’s menubars. Why? Because Java doesn’t give you a very good way to drop down to the native facilities whenever the abstraction fails. When you’re programming in AWT, you can’t figure out the HWND of a window, you can’t call the Microsoft APIs, and you certainly can’t intercept WM_PAINT and do it differently. And Sun made it plenty clear that if you tried to do that, you weren’t Pure. You were Polluted, and to hell with you.

After a number of highly publicized failures to build GUIs with Java (e.g. Corel’s Java Office suite and Netscape’s Javagator), enough people know to stay away from this world. Eclipse built their own windowing library from the ground up using native widgets just so they could write Java code that had a reasonably native look and feel.

The Mozilla engineers decided to address the cross platform problem with their own invention called XUL. So far, I’m impressed. Mozilla finally got to the point where it tastes like real food. Even my favorite bugaboo, Alt+Space N to minimize a window, works in Mozilla; it took them long enough but they did it.

Mitch Kapor, who founded Lotus and created 123, decided for his next application to go with something called wxWindows and wxPython for cross platform support.

Which is better, XUL, Eclipse’s SWT, or wxWindows? I don’t know. They are all such huge worlds that I couldn’t really evaluate them and tell. It’s not enough to read the tutorials. You have to sweat and bleed with the thing for a year or two before you really know it’s good enough or realize that no matter how hard you try you can’t make your UI taste like real food. Unfortunately, for most projects, you have to decide on which world to use before you can write the first line of code, which is precisely the moment when you have the least information. At a previous job we had to live with some pretty bad architecture because the first programmers used the project to teach themselves C++ and Windows programming at the same time. Some of the oldest code was written without any comprehension of event-driven programming. The core string class (of course, we had our own string class) was a textbook example of all the mistakes you could make in designing a C++ class. Eventually we cleaned up and refactored a lot of that old code but it haunted us for a while.

So for now, my advice is this: don’t start a new project without at least one architect with several years of solid experience in the language, classes, APIs, and platforms you’re building on. If you have a choice of platforms, use the one your team has the most skills with, even if it’s not the trendiest or nominally the most productive. And when you’re designing abstractions or programming tools, go the extra mile to make them leak proof.

Human Task Switches Considered Harmful

When you’re managing a team of programmers, one of the first things you have to learn to get right is task allocation. That’s just a five-dollar word for giving people things to do. It’s known colloquially as “file dumping” in Hebrew (because you dump files in peoples’ laps). And how you decide which files to dump in which laps is one of the areas where you can get incredible productivity benefits if you do it right. Do it wrong, and you can create one of those gnarly situations where nobody gets anything accomplished and everybody complains that “nothing ever gets done around here.”

Since this site is for programmers, I’m going to warm up your brains a little bit with a programming problem. 

Suppose you have two separate computations to perform, A and B. Each computation requires 10 seconds of CPU time. You have one CPU which, for the sake of this problem, doesn’t have anything else in the queue.

On our CPU, multitasking is optional. So you can either do these computations one after the other…

Sequential Processing

Computation A Computation B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

… or, you can multitask. If you multitask, on this particular CPU, tasks run for 1 second at a time, and a task switch takes no time at all.

Multitasking

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Which would you rather do?  Most people’s gut reaction is that multitasking is better. In both cases, you have to wait 20 seconds to get both of your answers. But think about how long it takes to get the results to each computation.

In both cases, the results of Computation B (shown in blue) take 20 seconds to arrive. But look at Computation A. With multitasking, its results take 19 seconds to arrive… yet with sequential processing they are ready in only 10 seconds.

In other words, in this nice contrived example, the average time per computation is lower (15 seconds rather than 19.5 seconds) when you do sequential processing rather than multitasking. (Actually, it’s not such a contrived example — it’s based on a real problem Jared had to solve at work).

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 seconds 15
Multitasking 19 seconds 20 seconds 19.5

Earlier I said that “a task switch takes no time at all.” Actually, on real CPUs, a task switch takes a little bit of time… basically enough time to save out the state of the CPU registers and load the CPU registers for the other task. Realistically, this is as close to negligible as possible. But to make life interesting, let’s imagine that task switches take half a second. Now things look even worse:

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 + 1 task switch =
20.5 seconds
15.25
Multitasking 19 + 18 task switches =
28 seconds
20 + 19 task switches = 29.5 seconds 28.75

Now … just humor me, I know this is silly … what if task switches take a whole minute?

Method Computation A takes Computation B takes Average
Sequential 10 seconds 20 + 1 task switch =
80 seconds
45 seconds
Multitasking 19 + 18 task switches =
1099 seconds
20 + 19 task switches = 1160 seconds almost 19 minutes!

The longer a task switch takes, the worse the multitasking penalty.

That, in and of itself, is not so earth shaking, is it? Pretty soon I’m going to be getting irate email from morons accusing me of being “against” multitasking. “Do you want to go back to the days of DOS when you had to exit WordPerfect to run 1-2-3?” they will ask me.

But that’s not my point. I just want you to agree with me that in this kind of example:

a) sequential processing gets you results faster on average, and

b) the longer it takes to task switch, the bigger the penalty you pay for multitasking.

OK, back to the more interesting topic of managing humans, not CPUs. The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That’s because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming. A programmer coding at full throttle is keeping zillions of things in their head at once: everything from names of variables, data structures, important APIs, the names of utility functions that they wrote and call a lot, even the name of the subdirectory where they store their source code. If you send that programmer to Crete for a three week vacation, they will forget it all. The human brain seems to move it out of short-term RAM and swaps it out onto a backup tape where it takes forever to retrieve.

How long? Well, my software company recently dropped what we were doing (developing a software product codenamed CityDesk) to help a client with a bit of an emergency situation for three weeks. When we got back to the office, it seemed to take another three weeks to get back to full speed on CityDesk.

On the individual level — have you ever noticed that you can assign one job to one person, and they’ll do a great job, but if you assign two jobs to that person, they won’t really get anything done? They’ll either do one job well and neglect the other, or they’ll do both jobs so slowly you feel like slugs have more zip. That’s because programming tasks take so long to task switch. I feel like when I have two programming projects on my plate at once, the task switch time is something like 6 hours. In an 8-hour day, that means multitasking reduces my productivity to 2 hours per day. Pretty dismal.

As it turns out, if you give somebody two things to work on, you should be grateful if they “starve” one task and only work on one, because they’re going to get more stuff done and finish the average task sooner. In fact, the real lesson from all this is that you should never let people work on more than one thing at once. Make sure they know what it is. Good managers see their responsibility as removing obstacles so that people can focus on one thing and really get it done. When emergencies come up, think about whether you can handle it yourself before you delegate it to a programmer who is deeply submersed in a project.

Two Stories

I want to tell you two stories from my career which I think are classic illustrations of the difference between tech companies that are well-managed and tech companies that are disasters. It comes down to the difference between trusting employees and letting them get things done, versus treating them like burger flippers that need to be monitored and controlled every minute, lest they wander off and sabotage everything.

My first assignment at my first job was working at Microsoft, where I was told to come up with a new macro language strategy for Excel. Pretty soon, I had the first draft of the “Excel Basic” spec (which later evolved into Visual Basic for Applications, but that’s another story). Somehow, this mysterious group of people at Microsoft called the “Application Architecture” group got wind of my spec, which must have concerned them, because for some reason they thought that they were in charge of things like macro language strategies, and they asked to see my spec.

I asked around. Who’s the Application Architecture group? Nobody seemed to think they were very serious. It turns out that they were a group of just four people, recent hires with PhDs (very unusual for Microsoft). I sent them a copy of my spec and went to meet them, in case they had something interesting to say.

“Blah blah!” said one of them. “Blah blah blah, blah blah blah!” said another. I don’t think they quite had anything interesting to say. They were very enamored of the idea of subclassing and sort of thought that people making macros in Excel wanted to subclass a lot of things. In any case, one of the fellows said, “Well, this is all very interesting. What’s next? Who has to approve your spec?”

I laughed. Even though I had only been at Microsoft for a few months, I knew that there was no such thing as somebody approving my spec. Hell, nobody had time to read my spec, let alone approve it. The programmers were bugging me every day to get them more pages so that they could write more code. My boss (and his boss) made it very clear to me that nobody else understood macros or had time to work on macros, so whatever I did, it better be right. And here this PhD working in a strange research group at Microsoft assumed that things were a bit more formal than that.

I pretty rapidly realized that the App Architecture group knew even less than I did about macros. At least, I had talked to a handful of macro developers and some Excel old-timers to get a grip on what people actually did with Excel macros: things like recalculating a spreadsheet every day, or rearranging some data according to a certain pattern. But the App Architecture group had merely thought about macros as an academic exercise, and they couldn’t actually come up with any examples of the kind of macros people would want to write. Pressured, one of them came up with the idea that since Excel already had underlining and double-underlining, perhaps someone would want to write a macro to triple underline. Yep. REAL common. So I proceeded to ignore them as diplomatically as possible.

This seemed to piss off a guy named Greg Whitten who headed up the App Architecture group. Now, Greg was something like Microsoft employee number 6. He had been around forever; nobody could quite point to anything he had done but apparently he had lunch with Bill Gates a lot and GW-BASIC was named after him. Greg called a BIG MEETING and proceeded to complain about how the Excel team (meaning me) was screwing up the macro strategy. We pressured him to come up with some specific reasons but his arguments just weren’t convincing. I thought it was nice that here I was, a new hire pipsqueak right out of college, arguing with employee number 6 and apparently winning the argument. (Can you imagine that happening at a Grey Flannel Suit company?) My programming team, headed by Ben Waldman (now a VP at Microsoft) backed me up completely, which was all that really mattered, because the programming team wrote the code and thus had the final say on how things got done.

I would have been perfectly happy to leave it at that. If the Apps Architecture team needed care and feeding and wanted to argue about stuff, that was OK, I would argue with them as much as they wanted as long as they left the programmers alone to do their work. But then something even more interesting happened that blew my mind. I was sitting at lunch with some coworkers, in the Redmond sun, when Pete Higgins came up to me. At that time Pete was the general manager for Office — I knew who he was, of course, but didn’t expect that he knew me very well. 

“How’s it going, Joel?” he asked. “I hear you’ve been having some issues with the App Architecture group.”

“Oh no!” I said. “Nothing I can’t handle.”

“Say no more,” he said, “I understand.” He left. By the next day the rumor had gotten back to me: the App Architecture group was disbanded. Not only that, but each member of the group was sent to a different department at Microsoft, as far apart as possible. I never heard from them again.

I was blown away, of course. At Microsoft, if you’re the Program Manager working on the Excel macro strategy, even if you’ve been at the company for less than six months, it doesn’t matter – you are the GOD of the Excel macro strategy, and nobody, not even employee number 6, is allowed to get in your way. Period.

This sends a really strong message. For one, it makes everyone that much more conscientious about their jobs. They can’t hide behind the idea that “management approved their spec,” since management really didn’t look too closely at their spec. All management did was hire smart people and gave them something to do. For another, it makes for an extremely nice place to work. Who doesn’t want to be king of their own domain? Software, by its nature, is very easy to divide into smaller and smaller components, so it’s always possible to divide up responsibility among people and let people own an area. This is probably THE  reason why software people love working at Microsoft.


Years passed. I found myself working at Juno, an online service and free email provider. This time, the experience was the exact opposite of my work at Microsoft. I had two programmers reporting to me, but my own manager constantly undermined my (limited) authority by going directly to my reports and giving them things to do, often without even telling me. Even for trivial requests like days off, my manager thought that it was his job to approve or disapprove the request.

After a couple of years at Juno I was working on the new user signup feature. For Juno 3.x, a major release, I was going to be in charge of a complete overhaul of the signup process. By this time, I was a relatively senior member of the technical team; I got great performance reviews, and my managers seemed to appreciate the work I was doing. But they just couldn’t bring themselves to trust me. Command and control.

One part of the signup process asked the users to type in their birthday. This was just one small bit of a lengthy signup process that went on for something like 30 screens as Juno grilled you about your income, your favorite sports, how many children you have and how old they were, and about 100 other things. To make the signup process a little bit easier, I wanted to change the birthday field to be free format, so you could type “8/12/74” or “August 12, 1974” or “12 Aug 74” or whatever. (Have you used Outlook? It would work like Outlook, where you could type dates in just about any format and it would accept them).

Without going into too much detail, my manager decided he didn’t like this. It became an issue of ego for him. First he yelled at the designer who was working on that page (without even telling me). Then he yelled at me. Then he reminded me every single day that I had to change it to the way he wanted it. Then he got the CEO of the company to review it, and made a big show out of getting the CEO of the company to criticize my new design. Even the CEO at Juno is perfectly happy to interfere in work done at the lowest level in the company, in fact, it’s standard operating procedure.

I was furious, needless to say. It was a small thing, a matter of taste, really. Some people would prefer my way. Some people would prefer his. In either case, the message was clear: you WILL do as you are told here, dammit. It was a very command-and-conquer mentality that was more of a battle of cojones than a discussion of user interface design.

I won’t say that this is the reason I left Juno, but it does illustrate the reason I left Juno: it was the idea that no matter how hard you work, no matter how smart you are, no matter whether you are ‘in charge’ of something or not, you have no authority whatsoever for even the tiniest thing. None. Take your damn ideas, training, brains, and intelligence, all the things we’re paying you for, and shove it. And at Juno, there were plenty of managers, something like 1/4 of all the employees, and so they had plenty of times to stick their fingers into every single decision and make sure that they were in control. The contrast with Microsoft, where VP’s descended from Building 9 to make it clear that you have the authority to get things done, was stark.

Hanging Tree in Jaffa, Israel

To some extent, Juno’s hopelessly inept management process is a factor of being a New York City company, not a West Coast company, so modern styles of management haven’t quite permeated. It’s also a problem caused by the deep inexperience of Juno’s managers, and it originates at the top – the CEO, a 29 year old who has never worked outside D. E. Shaw, who interferes in everything he can get his fingers into, including the wording on error messages that come up when things go wrong; the CTO regularly screams at his reports if they dare to question his wisdom; they take it out on the programmers, who go home and kick their dogs. Compare this to Microsoft, where things are done at the lowest level, and most managers act like their most important job is to run around the room, moving the furniture out of the way, so people can concentrate on their work.