31

I’ll be DJing at the party tonight.

What you see here:

  • Pioneer DJM-600 Professional DJ Mixer
  • Two Pioneer CDJ-800 Digital Vinyl Turntables (acts like a turntable but plays CDs)
  • Trusty Shure SM-58, mentioned earlier on this site
  • Sony MDR-V700DJ Studio Monitor Series DJ Headphones
  • Rotel RA-1060 Stereo integrated amplifier
  • B&W speaker system
  • Two IBM Thinkpad Laptops (one for visual effects and 2004 countdown; the other for playing MP3s)
  • Harman Kardon DVD 25 Progressive Scan DVD Player
  • (Offscreen) Pioneer 43″ HDTV Plasma Monitor showing visual effects generated from the audio track using G-Force Gold

22

I’ve been sanity-checking FogBUGZ for Unix by installing various OSes under VMware.

Easiest Linux to install: RedHat 9. Mandrake is not bad but still uses some jargon that makes it not quite ready for prime time, for example, you would not be able to get through setup without knowing what “root” means. SuSE went to a lot of trouble to create a good setup, then they go out of their way to make it difficult and slow to install if you don’t pay them for disks… my SuSE setup is still not done after several days of work. FreeBSD is pretty difficult to setup. Debian is very close to impossible, even for geeks.

The reason I need all these setups is because there are so many different ways to distribute software on Unix: we had to produce a .rpm, a .deb, a .tar.gz, and a .dmg for OS X.

(FogBUGZ for Unix system requirements: Unix, PHP 4, MySQL, Apache. It will ship in a few days).

Five versions of Unix on one box

15

This month’s Book of the Month is an excuse for me to write a long treatise about how modern software development is a world mostly divided into two large ideological cultures: The Culture of Unix Programming and The Culture of Windows Programming.

Biculturalism

By now, Windows and Unix are functionally more similar than different. They both support the same major programming metaphors, from command lines to GUIs to web servers; they are organized around virtually the same panoply of system resources, from nearly identical file systems to memory to sockets and processes and threads. There’s not much about the core set of services provided by each operating system to limit the kinds of applications you can create.

What’s left is cultural differences. Yes, we all eat food, but over there, they eat raw fish with rice using wood sticks, while over here, we eat slabs of ground cow on bread with our hands. A cultural difference doesn’t mean that American stomachs can’t digest sushi or that Japanese stomachs can’t digest Big Macs, and it doesn’t mean that there aren’t lots of Americans who eat sushi or Japanese who eat burgers, but it does mean that Americans getting off the plane for the first time in Tokyo are confronted with an overwhelming feeling that this place is strange, dammit, and no amount of philosophizing about how underneath we’re all the same, we all love and work and sing and die will overcome the fact that Americans and Japanese can never really get comfortable with each others’ toilet arrangements.

What are the cultural differences between Unix and Windows programmers? There are many details and subtleties, but for the most part it comes down to one thing: Unix culture values code which is useful to other programmers, while Windows culture values code which is useful to non-programmers.

This is, of course, a major simplification, but really, that’s the big difference: are we programming for programmers or end users? Everything else is commentary.

The Art of UNIX Programming - book coverThe frequently controversial Eric S. Raymond has just written a long book about Unix programming called The Art of UNIX Programming exploring his own culture in great detail. You can buy the book and read it on paper, or, if Raymond’s politics are just too anti-idiotarian for you to consider giving him money, you can even read it online for free and rest assured that the author will not receive a penny for his hard work.

Let’s look at a small example. The Unix programming culture holds in high esteem programs which can be called from the command line, which take arguments that control every aspect of their behavior, and the output of which can be captured as regularly-formatted, machine readable plain text. Such programs are valued because they can easily be incorporated into other programs or larger software systems by programmers. To take one miniscule example, there is a core value in the Unix culture, which Raymond calls “Silence is Golden,” that a program that has done exactly what you told it to do successfully should provide no output whatsoever. It doesn’t matter if you’ve just typed a 300 character command line to create a file system, or built and installed a complicated piece of software, or sent a manned rocket to the moon. If it succeeds, the accepted thing to do is simply output nothing. The user will infer from the next command prompt that everything must be OK.

This is an important value in Unix culture because you’re programming for other programmers. As Raymond puts it, “Programs that babble don’t tend to play well with other programs.” By contrast, in the Windows culture, you’re programming for Aunt Marge, and Aunt Marge might be justified in observing that a program that produces no output because it succeeded cannot be distinguished from a program that produced no output because it failed badly or a program that produced no output because it misinterpreted your request.

Similarly, the Unix culture appreciates programs that stay textual. They don’t like GUIs much, except as lipstick painted cleanly on top of textual programs, and they don’t like binary file formats. This is because a textual interface is easier to program against than, say, a GUI interface, which is almost impossible to program against unless some other provisions are made, like a built-in scripting language. Here again, we see that the Unix culture values creating code that is useful to other programmers, something which is rarely a goal in Windows programming.

Which is not to say that all Unix programs are designed solely for programmers. Far from it. But the culture values things that are useful to programmers, and this explains a thing or two about a thing or two.

Suppose you take a Unix programmer and a Windows programmer and give them each the task of creating the same end-user application. The Unix programmer will create a command-line or text-driven core and occasionally, as an afterthought, build a GUI which drives that core. This way the main operations of the application will be available to other programmers who can invoke the program on the command line and read the results as text. The Windows programmer will tend to start with a GUI, and occasionally, as an afterthought, add a scripting language which can automate the operation of the GUI interface. This is appropriate for a culture in which 99.999% of the users are not programmers in any way, shape, or form, and have no interest in being one.

There is one significant group of Windows programmers who are primarily coding for other programmers: the Windows team itself, inside Microsoft. The way they tend to do things is to create an API, callable from the C language, which implements the functionality, and then create GUI applications which call that API. Anything you can do from the Windows user interface can also be accomplished using a programming interface callable from any reasonable programming language. For example, Microsoft Internet Explorer itself is nothing but a tiny 89 KB program which wraps together dozens of very powerful components which are freely available to sophisticated Windows programmers and which are mostly designed to be flexible and powerful. Unfortunately, since programmers do not have access to the source code for those components, they can only be used in ways which were precisely foreseen and allowed for by the component developers at Microsoft, which doesn’t always work out. And sometimes there are bugs, usually the fault of the person calling the API, which are difficult or impossible to debug without the source code. The Unix cultural value of visible source code makes it an easier environment to develop for. Any Windows developer will tell you about the time they spent four days tracking down a bug because, say, they thought that the memory size returned by LocalSize would be the same as the memory size they originally requested with LocalAlloc, or some similar bug they could have fixed in ten minutes if they could see the source code of the library. Raymond invents an amusing story to illustrate this which will ring true to anyone who has ever used a library in binary form.

So you get these religious arguments. Unix is better because you can debug into libraries. Windows is better because Aunt Marge gets some confirmation that her email was actually sent. Actually, one is not better than another, they simply have different values: in Unix making things better for other programmers is a core value and in Windows making things better for Aunt Marge is a core value.

Picture of boats

Let’s look at another cultural difference. Raymond says, “Classic Unix documentation is written to be telegraphic but complete… The style assumes an active reader, one who is able to deduce obvious unsaid consequences of what is said, and who has the self-confidence to trust those deductions. Read every word carefully, because you will seldom be told anything twice.” Oy vey, I thought, he’s actually teaching young programmers to write more impossible man pages.

For end users, you’ll never get away with this. Raymond may call it “oversimplifying condescension,” but the Windows culture understands that end users don’t like reading and if they concede to read your documentation, they will only read the minimum amount, and so you have to explain things repeatedly… indeed the hallmark of a good Windows help file is that any single topic can be read by itself by an average reader without assuming knowledge of any other help topic.

How did we get different core values? This is another reason Raymond’s book is so good: he goes deeply into the history and evolution of Unix and brings new programmers up to speed with all the accumulated history of the culture back to 1969. When Unix was created and when it formed its cultural values, there were no end users. Computers were expensive, CPU time was expensive, and learning about computers meant learning how to program. It’s no wonder that the culture which emerged valued things which are useful to other programmers. By contrast, Windows was created with one goal only: to sell as many copies as conceivable at a profit. Scrillions of copies. “A computer on every desktop and in every home” was the explicit goal of the team which created Windows, set its agenda and determined its core values. Ease of use for non-programmers was the only way to get on every desk and in every home and thus usability über alles became the cultural norm. Programmers, as an audience, were an extreme afterthought.

The cultural schism is so sharp that Unix has never really made any inroads on the desktop. Aunt Marge can’t really use Unix, and repeated efforts to make a pretty front end for Unix that Aunt Marge can use have failed, entirely because these efforts were done by programmers who were steeped in the Unix culture. For example, Unix has a value of separating policy from mechanism which, historically, came from the designers of X. This directly led to a schism in user interfaces; nobody has ever quite been able to agree on all the details of how the desktop UI should work, and they think this is OK, because their culture values this diversity, but for Aunt Marge it is very much not OK to have to use a different UI to cut and paste in one program than she uses in another. So here we are, 20 years after Unix developers started trying to paint a good user interface on their systems, and we’re still at the point where the CEO of the biggest Linux vendor is telling people that home users should just use Windows. I have heard economists claim that Silicon Valley could never be recreated in, say, France, because the French culture puts such a high penalty on failure that entrepreneurs are not willing to risk it. Maybe the same thing is true of Linux: it may never be a desktop operating system because the culture values things which prevent it. OS X is the proof: Apple finally created Unix for Aunt Marge, but only because the engineers and managers at Apple were firmly of the end-user culture (which I’ve been imperialistically calling “the Windows Culture” even though historically it originated at Apple). They rejected the Unix culture’s fundamental norm of programmer-centricity. They even renamed core directories — heretical! — to use common English words like “applications” and “library” instead of “bin” and “lib.”

Raymond does attempt to compare and contrast Unix to other operating systems, and this is really the weakest part of an otherwise excellent book, because he really doesn’t know what he’s talking about. Whenever he opens his mouth about Windows he tends to show that his knowledge of Windows programming comes mostly from reading newspapers, not from actual Windows programming. That’s OK; he’s not a Windows programmer; we’ll forgive that. As is typical from someone with a deep knowledge of one culture, he knows what his culture values but doesn’t quite notice the distinction between parts of his culture which are universal (killing old ladies, programs which crash: always bad) and parts of the culture which only apply when you’re programming for programmers (eating raw fish, command line arguments: depends on audience).

There are too many monocultural programmers who, like the typical American kid who never left St. Paul, Minnesota, can’t quite tell the difference between a cultural value and a core human value. I’ve encountered too many Unix programmers who sneer at Windows programming, thinking that Windows is heathen and stupid. Raymond all too frequently falls into the trap of disparaging the values of other cultures without considering where they came from. It’s rather rare to find such bigotry among Windows programmers, who are, on the whole, solution-oriented and non-ideological. At the very least, Windows programmers will concede the faults of their culture and say pragmatically, “Look, if you want to sell a word processor to a lot of people, it has to run on their computers, and if that means we use the Evil Registry instead of elegant ~/.rc files to store our settings, so be it.” The very fact that the Unix world is so full of self-righteous cultural superiority, “advocacy,” and slashdot-karma-whoring sectarianism while the Windows world is more practical (“yeah, whatever, I just need to make a living here”) stems from a culture that feels itself under siege, unable to break out of the server closet and hobbyist market and onto the mainstream desktop. This haughtiness-from-a-position-of-weakness is the biggest flaw of The Art of UNIX Programming, but it’s not really a big flaw: on the whole, the book is so full of incredibly interesting insight into so many aspects of programming that I’m willing to hold my nose during the rare smelly ideological rants because there’s so much to learn about universal ideals from the rest of the book. Indeed I would recommend this book to developers of any culture in any platform with any goals, because so many of the values which it trumpets are universal. When Raymond points out that the CSV format is inferior to the /etc/passwd format, he’s trying to score points for Unix against Windows, but, you know what? He’s right. /etc/passwd is easier to parse than CSV,  and if you read this book, you’ll know why, and you’ll be a better programmer.

Craftsmanship

Making software is not a manufacturing process. In the 1980s everyone was running around terrified that Japanese software companies were setting up “software factories” that could churn out high quality code on an assembly line. It didn’t make any sense then and it doesn’t make sense now. Shoving a lot of programmers into a room and lining them up in neat rows did not really help get the bug counts down.

If writing code is not assembly-line style production, what is it? Some have proposed the label craftsmanship. That’s not quite right, either, because I don’t care what you say: that dialog box in Windows that asks you how you want your help file indexed does not in any way, shape, or form resemble what any normal English speaker would refer to as “craftsmanship.”

iPodWriting code is not production, it’s not always craftsmanship (though it can be), it’s design. Design is that nebulous area where you can add value faster than you add cost. The New York Times magazine has been raving about the iPod and how Apple is one of the few companies that knows how to use good design to add value. But I’ve talked enough about design, I want to talk about craftsmanship for a minute: what it is and how you recognize it.

I’d like to tell you about at a piece of code I’ve been rewriting for CityDesk 3.0: the file import code. (Advertainment: CityDesk is my company’s easy-to-use content management product.)

The spec seems about as simple as any snippet of code can be. The user chooses a file using a standard dialog box, and the program copies that file into the CityDesk database.

This turned out to be a great example of one of those places where “the last 1% of the code takes 90% of the time.” The first draft of the code looked like this:

  1. Open the file
  2. Read it all into a big byte array
  3. Store the byte array in a record

Worked great. For reasonable sized existing files it was practically instantaneous. It had a few little bugs, which I worked through one at a time.

The big bug surfaced when I stress-tested it by dragging a 120 MB file into CityDesk. Now, it is not common by any means for people to post 120 MB files on their web sites. In fact, it’s quite rare. But it’s not impossible, either. The code worked but took almost a minute and provided no visual feedback — the app just froze and appeared to be completely locked up. This is obviously not ideal.

From a UI perspective, what I really wanted was for long operations to bring up a progress bar of some sort, along with a Cancel button. In the ideal world you would be able to continue doing other operations with CityDesk while the file copy proceeded in the background. There were three obvious ways to do this:

  1. From a single thread, polling frequently for input events
  2. By launching a second thread and synchronizing it carefully
  3. By launching a second process and synchronizing it less carefully

My experience with #1 is that it never quite works. It is too hard to ensure that all the code throughout your application can be run safely while a file copy operation is in progress. And Eric S. Raymond has convinced me that threads are usually not as good a solution as separate processes: indeed years of experience have shown me that programming with multiple threads creates much additional complexity and introduces whole new categories of dangerously frightful heisenbugs. #3 seemed like a good solution, especially since our underlying database is multiuser and doesn’t mind lots of processes banging on it at the same time. So that’s what I’m planning to do when I get back from Thanksgiving vacation.

Notice, though, the big picture. We’ve gone from read the file/save it in the database to something significantly more complicated: launch a child process, tell it to read the file and save it in the database, add a progress bar and cancel button to the child process, and then some kind of mechanism so the child can notify the parent when the file has arrived so it can be displayed. There will also be some work passing command line arguments to the child process, and making sure the window focus behaves in an expected manner, and handling the case of the user shutting down their system while a file copy is in progress. I would guesstimate that when all is said and done I’ll have ten times as much code to handle large files gracefully, code that maybe 1% of our users will ever see.

And of course, a certain type of programmer will argue that my new child-process architecture is inferior to the original. It’s “bloated” (because of all the extra lines of code). It has more potential for bugs, because of all the extra lines of code. It’s overkill. It’s somehow emblematic of why Windows is an inferior operating system, they will say. What’s all this about progress indicators? they sneer. Just hit Ctrl+Z and then “ls -l” repeatedly and watch to see if the file size is growing!

A picture of the new door

The moral of the story is sometimes fixing a 1% defect takes 500% effort. This is not unique to software, no sirree, now that I’m managing all these construction projects I can tell you that. Last week, finally, our contractor finally put the finishing touches on the new Fog Creek offices. This consisted of installing shiny blue acrylic on the front doors, surrounded by aluminium trim with a screw every 20 cm. If you look closely at the picture, the aluminium trim goes all the way around each door. Where the doors meet, there are two pieces of vertical trim right next to each other. You can’t tell this from the picture, but the screws in the middle strips are almost but not exactly lined up. They are, maybe, 2 millimeters off. The carpenter working on this measured carefully, but he was installing the trim while the doors were on the ground, not mounted in place, and when the doors were mounted, “oops,” it became clear that the screws were not exactly lined up.

This is probably not that uncommon; there are lots of screws in our office that don’t line up perfectly. The problem is that fixing this once the holes are drilled would be ridiculously expensive. Since the correct placement for the screws is only a couple of millimeters away, you can’t just drill new holes in the door; you’d probably have to replace the whole door. It’s just not worth it. Another case where fixing a 1% defect takes 500% effort, and it explains why so many artifacts in our world are 99% good, not 100% good. (Our architect never stops raving about some really, really expensive house in Arizona where every screw lined up.)

It comes down to an attribute of software that most people think of as craftsmanship. When software is built by a true craftsman, all the screws line up. When you do something rare, the application behaves intelligently. More effort went into getting rare cases exactly right than getting the main code working. Even if it took an extra 500% effort to handle 1% of the cases.

Craftsmanship is, of course, incredibly expensive. The only way you can afford it is when you are developing software for a mass audience. Sorry, but internal HR applications developed at insurance companies are never going to reach this level of craftsmanship because there simply aren’t enough users to spread the extra cost out. For a shrinkwrapped software company, though, this level of craftsmanship is precisely what delights users and provides longstanding competitive advantage, so I’ll take the time and do it right. Bear with me.

01

Language Policy (book by Bernard Spolsky)Mazel Tov to the elder Spolsky on his latest book Language Policy. No, not computer languages.

Craftsmanship

Writing code is not production, it’s not always craftsmanship (though it can be), it’s design. Design is that nebulous area where you can add value faster than you add cost. The New York Times magazine has been raving about the iPod and how Apple is one of the few companies that knows how to use good design to add value. But I’ve talked enough about design, I want to talk about craftsmanship for a minute: what it is and how you recognize it.

Summer Internships

Are you a college student looking for a summer internship in software development? Fog Creek Software is the place for you! Details…

30

I spent the long weekend grinding through the backlog of Joel on Software translations. There are a bunch of new articles in various languages including new sections for Esperanto and Greek. All in all there are 264 translations in progress in 32 languages thanks to 242 volunteers around the world. 177 translations are complete and have already been posted.

There are a few articles, already translated, which just need copy editors before I can post them. If you read and write one of these languages fluently and are willing to help out, I’d really appreciate it! What’s involved is just looking for typos and errors and improving the translation wherever possible. If I don’t find anyone to edit the articles I will probably just go ahead and post them unedited but it would be nice to have a second set of eyes improving the quality of the translations.

Languages I need editors for: Chinese (Simplified), Chinese (Trad), Esperanto, Estonian, French, Greek, Hungarian, Indonesian, Japanese, Korean, Portuguese (Port.), Romanian, Russian, Swedish, and Tamil.

A frequently asked question: why bother with these translations? Surely any real programmer knows English! And my frequently answered answer: First of all, not every programmer knows English, and if they do, they may not know it that well, so they may not really enjoy reading things written in English if they don’t have to. Second, even if the programmers have learned enough English to decipher online documentation, their pointy-haired bosses from management may not have.

Another frequent question: why not just use Babelfish or Google Language Tools or another similar translation tool? Answer: They are seriously little. You cannot include/understand simply the exit. Er, what I meant to say was, they are seriously inadequate. The quality of translations produced by automatic software is so horrible that you really can’t understand the output. Try asking Google to translate http://french.joelonsoftware.com from French to English for some real howlers. “Then why does nobody make planning? Two principal reasons. Firstly, it is really difficult. Secondly, nobody believes that that is worth the sorrow of it. Why give so much difficulty to be worked on a planning if it is known that it will not be correct?”

14

Time for the next Book of the Month.

Almost any argument about managing the software development process inevitably deteriorates into anecdote-ping-pong. “We did wawa and everyone quit.”

“Oh yeah? Then how do you explain Company X? They wawa regularly and their stock is up 20%!”

If you have even the slightest bit of common sense, you should ask: “Where’s the data? If I’m going to switch to Intense Programming I want to see proof that the extra money spent on dog kennels and bird cages is going to pay for itself in increased programmer self-esteem. Show me hard data!”

And, of course, we have none.

One set of people will tell you you gotta have private offices with walls and a door that closes. Another set of extremos will tell you everyone has to be in a room together, shoulder-to-shoulder. Neither of them have any hard data whatsoever, where by “hard data” I mean “data that wouldn’t be laughed out of a sixth-grade science classroom.” The truth is, you can’t honestly compare the productivity of two software teams unless they are trying to build exactly the same thing under exactly the same circumstances with the exact same human individuals, who have been somehow cloned so they don’t learn anything the first time through the experiment.

Tom DeMarco was so frustrated at the inherent impossibility of providing any kind of hard data that he went so far as to write a novel in which he fantasizes about a bizarre land in which programmers are so cheap you actually can do experiments where, say, half the people have offices and half the people have cubicles.

But we don’t have the data. We don’t have any data. You can give us anecdotes left and right about how methodology X worked or didn’t work, but you can’t prove that when it worked it wasn’t just because of one really, really good programmer on the team, and you can’t prove that when it failed is wasn’t just because the company was in the process of going bankrupt and everybody was too demoralized to do anything at all, Aeron chairs notwithstanding.

Facts and Fallacies of Software Engineering CoverBut don’t give up hope. We do have the collective wisdom of fifty years of building software to draw from. Or at least, it’s somewhere. Your typical startup with three pals from college may not exactly have the collective wisdom, so they’re going to reinvent things from scratch that IBM figured out in 1961, or go bankrupt failing to reinvent them. Too bad, because they could have read Facts and Fallacies of Software Engineering, by Robert L. Glass, the best summary of what the software profession should have agreed upon by now. Here are just a few examples from the 55 facts and 10 fallacies in the book:

  • The most important factor in software work is not the tools and techniques used by the programmers, but rather the quality of the programmers themselves.
  • Adding people to a late project makes it later.
  • Reuse-in-the-small (libraries of subroutines) began nearly 50 years ago and is a well-solved problem.
  • Reuse-in-the-large (components) remains a mostly unsolved problem, even though everyone agrees it is important and desirable.

You can read the others in the table of contents on Amazon. One of the best things about the book is that it has sources for each fact and fallacy, so you can go back and figure out why we collectively believe that, say, code inspection is valuable but cannot and should not replace testing. This is bound to be particularly helpful when you need ammunition for your arguments with people in suits making absurd demands (“Can we make a baby in 1 month if we hire 9 mothers?”).