“Programmers should only use lower level tools for those parts of the product where they are adding the most value. For example, if you’re writing a game where the 3D effects are your major selling point, you can’t use an off the shelf 3D engine, you have to roll your own. But if the major selling point of your game is the story, don’t waste time getting great 3D graphics — just use a library.”
For your weekend reading pleasure, Rick Chapman interviewed me.
CityDesk Is Here!
A few last-minute details, then — CityDesk is shipping! People can buy it! It’s a great feeling after a lot of work.
Time to say thank you. First and foremost, thanks to the CityDesk team who made this release happen on time. Thanks to our beta testers, for bearing with us through thick and thin and five beta releases, for trying CityDesk and building your sites with it. Thanks to our families and friends for supporting us while we worked late. Thanks to you, Joel on Software readers, for bearing with me while I stopped writing about software in the abstract and spent some time doing it. It’s been an exciting period and I’m going to sleep for a few months.
Do you have a T3?
The starter edition of CityDesk is free. But you can’t download the free version yet because our office T1 can’t handle the load and it will take a while to get the software up on Tucows.
In the meantime, we’re looking for a volunteer with a lot of extra bandwidth who is willing to mirror the free version for a few weeks until we have a permanent server for it with lots of bandwidth. Please email me if you can do this.
The SETUP is 8 MB and we are expecting thousands of downloads in the first few days — so don’t volunteer unless you really have a ton of excess bandwidth that you’d be willing to donate 🙂 Thanks.
Cool, We Got Mentioned In Australia
Sydney’s IT News: “Most of all, Spolsky wants people to want his product. He writes: ‘My number one priority is to make an application that’s easy to use so that people like it, so that they tell their friends about it, so that everybody buys it and we have to hire full-time employees just to slit open the envelopes and take out the cheques.'” Actually, I think I said “checks!”
The last few days before shipping a product should be extremely quiet, as the rate of incoming bugs approaches zero. That’s where we are now. I’ve been putting the final touches on a new version of Fog Creek’s website (you can see a sneak preview here) and taking the word “Beta” out of the code wherever it appears. We’re likely to go live this week — exactly according to the schedule we set for ourselves six months ago.
Happy Thanksgiving!
I can’t say enough nice things about VMWare. This program has been amazingly helpful during the last few weeks as we tried to get CityDesk to work on every known version of 32-bit Windows. I have set up dozens of virtual machines, everything from a simple DOS partition (helpful as a starting point for installing other OSs), a bunch of combinations of NT 4.0, Chinese and Hebrew Win2K (even though our program is in English and doesn’t do anything fancy, we had various bugs that were revealed on these systems), assorted versions of Win 95/98/Me going all the way back to the August, 1995 release, even a small network of machines with a primary domain controller which we used for testing FogBUGZ setup.
Getting code to work on the entire universe of Windows machines is a lot of work. That’s the real appeal of “write once, run anywhere” systems like Java. In theory, if you use the Java Virtual Machine, the burden is on the VM vendor to provide compatibility with all these platforms. In reality, as Java programmers learned, code is just too fragile for this to work very well. When I developed a game with Java I learned that Java’s inability to guarantee exactly when threads would run (a seemingly harmless concession to the fact that CPU scheduling is basically unpredictable) actually meant that on the Macintosh, some threads got starved, um, forever, basically, until the other threads tried to do i/o in fact, which is not what I had assumed and made my game not very challenging on Macs. (This was in 1996. Don’t email me with workarounds, fixes, or to say that this bug has been fixed.)
Yesterday’s Bug Du Jour is an example of the kind of thing that trips people up. Michael was allocating some memory using the ancient Windows API GlobalAlloc. Later, he was calling a function GlobalSize to determine the size of that memory. On our development systems (Windows 2000) GlobalSize returns the same value as the block you allocated. Allocate 13 bytes, and GlobalSize will return 13.
We got a bug report that “Copy and Paste don’t work” from a user with Windows 98. As you see from the screenshot above, I’ve got a Windows Me VM set up with VB6. Stepping through the code in the debugger I noticed the GlobalSize call, and remembered from the days of Win 95 that GlobalSize used to return the size of the actual allocated block, which was larger than you allocated and usually a multiple of 64. This led to the bug.
Now, the programmer at Microsoft who changed the behavior of GlobalSize probably thought he wasn’t breaking anything. The documentation for the function “GlobalSize” says clearly, “the size of a memory block may be larger than the size requested when the memory was allocated.” In fact the Microsoftee probably thought it was a minor and harmless improvement to GlobalSize to have it always return the size that you requested. Obviously any old code, that doesn’t trust the return value from GlobalSize, will continue to work. So why not improve the function?
Not every programmer studies every line of the documentation for every function they use, and if the code is working, they tend to move on to something else. And it’s not like everything is documented all that well — the type of minutiae that I’m talking about here rarely gets discussed in documentation. And this is where you get these issues. The rest of the world discovered this problem, long the bane of WINE programmers, when the second web browser shipped, and suddenly everybody was noticing how the bugs they had counted on to make their pages look kewl weren’t there any more. As we speak zillions of HTMLers are bemoaning the fact that IE6 now adheres to the standard for what <CENTER> does to the text inside a table, and so their pages that looked left-justified look like wedding invitations in IE6.
How did we get into this mess? Larry Wall famously said, “People understand instinctively that the best way for computer programs to communicate with each other is for each of the them to be strict in what they emit, and liberal in what they accept.” I think that the evolution of HTML has proven that this isn’t such a great idea. In fact, the stricter the API is about its input, the more likely the code is going to work in funny situations. The designers of Java got it right when they decided that nothing about the Java spec should leave any choice to the compiler developers (at least, not in the gratuitous way that C did, where the size of basic data types was not fixed). A better quote comes from Russian Field Marshal Suvorov: “A hard drill makes an easy battle.” You want your compiler and your development environment to be as strict as possible; you want it to literally generate random return values for GlobalSize so that you don’t get into the habit of counting on something that won’t be there everywhere; you want to use French international settings on Chinese Windows 2000 with an absurd color scheme, DVORAK keyboard, trackball, 640×480 VGA mode, and huge ugly fonts on your development system so that you remember to bake in the code that adjusts for all these things. Then your application will be buff and strong and it will laugh in the face of wimpy problems like people who use commas instead of dots as the decimal. Ha. I eat commas for breakfast, your code will say, with a Russian accent.
Anyway, this is what it takes to get software that works on hundreds of millions of computers. Those of you who develop apps that only have to run on one computer or in a controlled environment have it easy, but you’re getting flabby. One of these days you’ll need to get it to run on a second computer and you’ll need to pull an all nighter, installing a complete development environment on that computer and debugging for two hours, before you discover that you didn’t account for the possibility of spaces in the installation file path because the first computer didn’t have them.
Hopefully in the future the concept of a Virtual Machine, whether it’s Java, .NET, or something else, will alleviate this pain, but we ain’t there yet. For now I’m happy that with ten minutes of debugging, I can make my app work for all the people who like pink on orange text and set up Windows accordingly. We spent probably 3 weeks of work, out of a 1 year development cycle, fixing configuration bugs. A small price to pay to increase the size of our potential customer base from just US Windows 2000 to the entire universe of NT 4, 95, 98, Me, and XP, worldwide. Cool.
Larry Wall famously said, “People understand instinctively that the best way for computer programs to communicate with each other is for each of the them to be strict in what they emit, and liberal in what they accept.” I think that the evolution of HTML has proven that this isn’t such a great idea. In fact, the stricter the API is about its input, the more likely the code is going to work in funny situations. The designers of Java got it right when they decided that nothing about the Java spec should leave any choice to the compiler developers (at least, not in the gratuitous way that C did, where the size of basic data types was not fixed). A better quote comes from Russian Field Marshal Suvorov: “A hard drill makes an easy battle.” You want your compiler and your development environment to be as strict as possible; you want it to literally generate random return values for GlobalSize so that you don’t get into the habit of counting on something that won’t be there everywhere; you want to use French international settings on Chinese Windows 2000 with an absurd color scheme, DVORAK keyboard, trackball, 640×480 VGA mode, and huge ugly fonts on your development system so that you remember to bake in the code that adjusts for all these things. Then your application will be buff and strong and it will laugh in the face of wimpy problems like people who use commas instead of dots as the decimal. Ha. I eat commas for breakfast, your code will say, with a Russian accent.
A Hard Drill Makes an Easy Battle
I noticed that people were putting HTML tags in discussion group postings, and they didn’t know what to do about URLs. So I added some tiny text below the edit box where you type your posting… not that anybody reads tiny text, but we’ll see if it helps!
I’ve long believed that tiny changes in the user interface of any kind of community software result in dramatic changes in the character of the community that you create. I noticed this way back in college, when CompuServe forums generally had two-line postings and nobody ever quoted other people, but Usenet newsgroups had lengthy postings with 21-line ASCII art signatures and 50% quoting. Usenet resulted in the ultimate pathology: people quoting an entire three page article, and inserting their tedious nitpicks between every two lines.
My own discussion software does not have threading. “Threading” is technical jargon for a discussion feature where different people can branch in different directions by replying to replies. You end up with a tree of conversation. Most forum software has this feature and some people were rather angry that mine doesn’t.
I first noticed the value of one-train topics using the echo community software, which is, in all other respects, excruciatingly bad. Something interesting happens sociologically when you don’t have threading: the conversation is forced along one train of thought. People feel like they can respond to the original inquiry, or they can respond to the last post, but if they want to nitpick about the third post and there are already twenty more posts after that, it’s just too late. This seems to eliminate the most gratuitous trivial “yes, I agree” or “you nitwit, you misspelled that” kind of posting which makes it boring to read other forums. The other thing which eliminates them is that there’s no automatic mechanism to quote other people. Sheer laziness will prevent most people from tagging on comments that aren’t an interesting extension to the current thread. If you really want to set off on a tangent, you have to write something like “I’d like to go back to what X said earlier…” which actually makes the whole topic read more coherently.
Another mechanism for eliminating trivial and uninteresting postings is the slashdot method of having moderators hide them. Theoretically, you can set your threshold high on slashdot so you only see the most interesting posts, but this gives the conversation a choppy feel when you’re reading it.
TechInterview
Michael’s TechInterview is back up and running, with its own discussion group. He’s moved it from Manila to CityDesk, which allowed us to completely decommission the Manila server.
Books
It’s about time that I updated my crufty old list of recommended books. What books should I have up there?
Ground rules: to keep this conversation on track, everybody gets to vote for ONE book, and one book only. I really want to hear what people think is the SINGLE best book on “painless software management.”
CityDesk, Part Five
Now we’re thanking our lucky stars that we didn’t have a bunch of stupid venture capitalists forcing us to copy all the other content management companies, and we’re grateful that we’re not in Silicon Valley where everyone meets at Bucks and Stanford University seminars and copies each other’s bad ideas, because the one thing we’ve heard from everybody who’s tried CityDesk, consistently, is that CityDesk is the easiest content management software they’ve ever seen, full stop. And we got this ease-of-use because we believed certain things about software.
Working on CityDesk, Part Five.
Alan Cooper: “My advice to Microsoft is to abandon the browser. The browser is a red herring; it’s a dead end. The idea of having batched processing inside a very stupid program that’s controlled remotely is a software architecture that was invented about 25 years ago by IBM, and was abandoned about 20 years ago because it’s a bad architecture. We’ve gone tremendously retrograde by bringing in Web browsers… We have stepped backward in terms of user interface, capability, and the breadth of our thinking about what we could do as a civilization. The browser is a very weak and stupid program because it was written as essentially a master’s thesis inside a university and as an experiment….”
Alan invented Visual Basic, which is the language we used for CityDesk, so it’s no big coincidence that I agree with him so wholeheartedly. And it’s true: from a UI architecture perspective, browsers are CICS + fonts. CICS is an extremely ancient interactive system used with IBM mainframe terminals. Basically, the server sends a text-mode, full-screen form down to the “smart” terminal. The user fills in the form and sends it back to the server. And that’s about all you can do from a UI perspective. The mainframe is happy because it can basically be about as stupid as a web server: queue up all incoming requests and deal with them sequentially. The user suffers because the UI is abysmal and the programmer suffers because you can’t MAKE a good UI no matter how hard you try.
As promised I’ve been writing a series of articles about the creation of CityDesk. The primary purpose of these articles is to share the reasoning behind some of the decisions we made as a software company, and the techniques we used in developing our flagship software product. Today I want to talk about the five guiding principles that shaped the architecture.
One of the biggest decisions we made about CityDesk was to make a standalone Windows executable. With all the hubbub about web services, not to mention the noisy Internet dotcom boom we just lived through, this was a non-obvious decision. Why is CityDesk one of those old-fashioned programs you download and run, instead of, say, a web site you go to, in the style of Blogger or Atomz? Or a web server you install, like the Big Iron content management systems (Interwoven, Vignette StoryServer, etc.)?
An even more non-obvious decision was not to use XML. People who were a little bit too close to Silicon Valley groupthink looked at us like we were crazy because CityDesk is not based on XML and is not delivered through a web browser.
Now we’re thanking our lucky stars that we didn’t have a bunch of stupid venture capitalists forcing us to copy all the other content management companies, and we’re grateful that we’re not in Silicon Valley where everyone meets at Bucks and Stanford University seminars and copies each other’s bad ideas, because the one thing we’ve heard from everybody who’s tried CityDesk, consistently, is that CityDesk is the easiest content management software they’ve ever seen, full stop. And we got this ease-of-use because we believed certain things about software.
1. You can’t do good UIs in a web browser.
When we started building CityDesk we kept hearing one thing about Big Iron content management systems, which were all web-browser-based. Despite the fact that these systems had web-based editing interfaces and advanced workflow, we kept hearing that in real life, reporters created their content in Microsoft Word, did the workflow the old way (email), and only at the very last minute, a secretary opened the Word file and cut and pasted it into the content management system.
Why?
Because nobody wants to compose in a big TEXTAREA on an HTML page.
There are too many things that are impossible to deliver properly in a web browser. First of all, there’s the latency. Even if the web server is running on your own machine, round trips still take a certain amount of time, so all web applications feel somewhat sluggish. Second, there is no decent text editing widget for web browsers. With CityDesk as a Windows program, I can give you a feature where you can drag a picture from the desktop into an article. There is no way to create that kind of user interface through the web. I can keep a word count in the corner of the screen and update it in the background whenever you stop typing. You can use Ctrl+S to save without losing your place in the document — something many writers have learned to do regularly. (Good luck creating a word processor inside a web browser that doesn’t instantly lose everything, without prompting, if the user closes the browser). CityDesk has menus. Remember menus? And they work exactly like you expected them to, because the web browser doesn’t have it’s own menu, with a bunch of irrelevant commands, that wants to eat the Alt key. We don’t have to waste any screen real estate on browser geegaws (like “back” buttons and spinning e-globes) that have no meaning for CityDesk. We get our own icon in the taskbar instead of looking like all the other web browsers you have open. I could go on for days about the nice UI things we can do in a Windows application that just can’t be done with a web browser. In fact there’s a whole chapter about it in the printed edition of my book.
“Sure,” you say, “you can’t get as slick an app through the web, but you get ubiquity! People can go to an Internet cafe in Phuket and …” Yeah, OK, that sounds great for Hotmail. And in the future we will build a typical compromise-laden web interface to CityDesk for those times when you’re at the beach in Thailand. In the meantime, my number one priority is to make an application that’s easy to use so that people like it, so that they tell their friends about it, so that everybody buys it and we have to hire full time employees just to slit open the envelopes and take out the checks.
2. XML is a Dumb Format For Storing Data
I’m not sure why XML got so sexy. It has its advantages; it’s sure a good idea for data interchange or for all those little files you need to store settings. But for real work it just can’t do what a solid, multiuser, relational database can do. The next time some uninformed analyst at Gartner or Giga or Forrester tells you “in the future, everything will be XML,” ask them how to do “SELECT author FROM books” fast with XML. Hint: you can’t. It has to be slow. XML is not the way to store a lot of data. Now tell me how to insert a new book at the beginning of the table without massive bitblts. Of course, I doubt if there is an analyst in one of those companies who would even understand that sentence, but that’s life. In the meantime, CityDesk uses a relational database. Over and out.
3. Most People Don’t Control Their Web Servers
Of all the people in the world who need to put data up on the web, only a tiny percentage of them have any control whatsoever over what software runs on their web server. We decided that CityDesk has to be an all-client application because probably 95% of the people who maintain web sites simply do not have the ability to install a content management system on their web server, even if they wanted to. We reduced the server requirements of CityDesk to (a) any web server that knows how to serve static files and (b) ftp or file copy access to that server. We think this will open up content management to a whole new universe of potential users.
4. If You Can Only Do One Platform, Do Windows.
I’d love to have a Mac version and a Linux version, but they are not good uses of limited resources. Every dollar I invest in CityDesk Windows will earn me 20 times as many sales as a dollar invested in a hypothetical Mac version. Even if you assume that Mac has a higher percentage of creative and home users, I’m still going to sell a heck of a lot more copies on Windows than I could on Mac. And that means that to do a Mac version, the cost had better be under 10% of the cost of a Windows version. Unfortunately, that’s nowhere near true for CityDesk. We benefit from using libraries that are freely available on Windows (like the Jet multiuser ACID database engine and the DHTML edit control) for which there are no equivalents on the Macintosh. So if anything, a Mac port would cost more than the original Windows version. Until somebody does something about this fundamental economic truth, it’s hard to justify Mac versions from a business perspective. (Incidentally, I have said time and time again, if Apple wants to save the Mac, they have to change this equation.)
And don’t get me started about Linux. I don’t know of anyone making money off of Linux desktop software, and without making money, I can’t pay programmers and rent and buy computers and T1s. Despite romantic rhetoric, I really do need to pay the rent, so for now, you’re going to have to rely on college kids and the occasional charitable big company for your Linux software.
5. Content Management Can’t Require Programmers
One of the reasons Big Iron content management costs so much is the team of programmers it takes to figure out the thing and set it up. Even the free and low-cost content management systems are designed by geeks for geeks. People have said to me that there’s no market for CityDesk because “with XML and XSL it’s a solved problem.”
Uh huh. Thanks. Even I can’t figure out XML and XSL, and I’m pretty sharp.
We decided that everything in CityDesk has to be simple enough for web designers and HTMLers, who are not necessarily programmers. And for the end-user, even HTML is too much. To the end-user CityDesk can’t be any more complicated than a WYSIWYG word processor.
The rule of thumb with ease-of-use is that if you make your program 10% easier, you’ll double the potential number of users of your product. In CityDesk we made absolutely no compromises on ease of use. Sure, it’s not the most powerful system out there. We’ll compromise on that. And you can’t update your blog from an Internet Cafe in Bali. Another compromise. But I guarantee you that any HTMLer can create CityDesk sites, and anyone who can use a word processor can manage a site that was created for them by an HTMLer, and you’ll never need a programmer.
Summary
All in all we made a bunch of decisions based on a world view that, frankly, was not very popular during the Internet boom. There was a time when you couldn’t get funding for a company that wasn’t a web pure play. And the lemming VCs thought that they had to do what all the other VCs were doing: pure web interfaces. To which I say, Thank you! Thank you for keeping your dumb overfunded companies out of my way, because now, over here in GUI land, I can’t find any real competition, and now you’re not funding anyone, so it’s smooth sailing for Fog Creek.
Prakash wants to know “how all of the esteemed contributors in this discussion list came across your website.”
CityDesk Beta
Last night at around 10 PM we fixed the last (known) bug in CityDesk Beta 2. This morning we’re going to do some smoke tests and last-minute banging, and if all goes well, we’ll release Beta 3 today.
For this beta we found, and fixed, some of the nastiest, hairiest, scariest bugs that we thought we’d never be able to fix. A mysterious crash that couldn’t be reproduced reliably. Occasional mauling of HTML code by the WYSIWYG editor, and even more occasionally completely destroying HTML code. (That bug, incidentally, was in a Microsoft library function, which we reimplemented ourselves.) Mysterious startup crashes when you’re using a non-Latin1 version of Windows (Hebrew, Greek, etc.) Inability to run as a normal user on Windows NT. And we sped up incremental publishing dramatically by caching CRCs.
Update, 7 PM. We managed to release Beta 3 today – four times! Scary little show-stopper bugs kept popping up. I’m praying that the rate at which show-stopper bugs keep popping up goes down dramatically so I can take the whole weekend off for a change. Fog Creek Software: “We Make Lots of Bugs, But At Least We Fix Them!”