Let That Be a Lesson To You, Son: Never Upgrade.

(Update: This piece originally ran on April Fools' day; although the content of the post is not an April Fools' joke, the retro styling definitely was. View a screenshot of how this post looked on April 1, 2008)

I occasionally follow Jamie Zawinski's blog. Jamie's an interesting guy. In the process of researching an earlier post, I discovered that he played a significant role in unearthing the classic Worse is Better paper:

About a year later [1991] we hired a young kid from Pittsburgh named Jamie Zawinski. He was not much more than 20 years old and came highly recommended by Scott Fahlman. We called him "The Kid." He was a lot of fun to have around: not a bad hacker and definitely in a demographic we didn't have much of at Lucid. He wanted to find out about the people at the company, particularly me since I had been the one to take a risk on him, including moving him to the West Coast. His way of finding out was to look through my computer directories - none of them were protected. He found the EuroPAL paper, and found the part about worse is better. He connected these ideas to those of Richard Stallman, whom I knew fairly well since I had been a spokesman for the League for Programming Freedom for a number of years. JWZ excerpted the worse-is-better sections and sent them to his friends at CMU, who sent them to their friends at Bell Labs, who sent them to their friends everywhere.

Or, perhaps you've read the classic Teach Yourself Programming in Ten Years? That was written by Peter Norvig, who is now the director of research at Google. It refers to Mr. Zawinski thusly:

One of the best programmers I ever hired had only a High School degree; he's produced a lot of great software, has his own news group, and made enough in stock options to buy his own nightclub.

I think you'll agree that it's fair to call Jamie Zawinski a world class software engineer. Jamie's blog documents, in great detail, how he runs his DNA Lounge club in San Francisco. It's a great read, full of fascinating, often geeky backstage details. The DNA Lounge is powered by open source software, including various flavors of Linux. Sometimes this can be painful. In 2006, Jamie ran into serious problems with the Linux sound architecture:

You may have noticed that the audio archives have only had one channel for the last few weeks. You would probably assume that's a simple matter of replacing a cable; turns out, not. As far as we can tell, the audio going into the computer is stereo, and somewhere in there, it drops (most of) the right channel. So, bad connector, right? No, we've tried four different sound cards, and checked the mixer settings. At this point it seems like the last time we (accidentally) upgraded ALSA, it introduced some software bug that is making one channel go away. I can't even fathom how such a bug could exist, but that's Linux for you.

We seem to have solved the "missing right channel" problem. It was, in fact, a software problem. We were running Fedora 4, and when we installed the latest patches on March 31, that's when the right channel vanished. We tried downgrading to the version of the kernel and ALSA as of three months ago, and that didn't fix it. But, Jonathan took all the sound cards home and tried them in his machine, and they all worked fine there. He was running Fedora 5. So we upgraded to that, and the problem went away.

That's right: upgrading to the latest FC4: breaks the world. Giving up on FC4 and going to FC5: un-breaks it. Nicely done, guys.

For years I've had it drummed into my head that you always have to keep your systems patched, if you aren't running the latest security fixes, the script kiddies will eat you alive, running a six month old OS is like leaving your front door wide open, blah blah blah. Well you know what? F**k that noise. I'm done upgrading anything ever. The next time I get this s**t into a state that seems even remotely stable, I'm never touching it again. If we get hacked, oh well. I have backups. It has got to be less work to recover from than constantly dealing with this kind of nonsense.

The DNA lounge provides streaming audio and video webcasts of whatever is going on any time the club is open. So problems like this are especially troubling -- Jamie's business depends on this stuff working.

I was particularly disturbed to find this recent entry:

I spent a solid four days trying to upgrade the kiosks from Red Hat 9 + LTSP 4.3 (vintage 2003) to... something newer. In this case, Ubuntu 10.7 + LTSP 5, since it seems like that's what the cool kids are running these days. Why would I do such a thing? Well, one reason is that the Firefox 3 beta would neither install nor compile on RH9 (missing libraries), and another was that the kiosks are a little crashy (they reboot themselves pretty regularly for no adequately explored reason), and also, it's "just kinda old", which some people will tell you might mean, maybe, kinda, less secure. So I figured I'd give it a shot.

Well, since this is not my first rodeo, when I say "upgrade" what I really mean is "do a fresh install on a spare drive."

So, after four days of this nonsense, I gave up, and just put the old drive back in. "Nonsense" in this case is defined as: the upgrade made the machines be even crashier than before (they can barely stay up for an hour) and it's a far worse kind of crashy: it's the kind of crashy where you have to press the shiny red button to make them come back to life, instead of them being able to do that themselves.

So, f**k it. They'll be running a 2003 version of Linux forever, because I frankly have better things to do with my time.

I can't fault Jamie's approach. A clean install of an operating system on a new hard drive -- for kiosks running controlled hardware, no less -- that's as good as it gets.

Apparently, Linux is so complex that even a world class software engineer can't always get it to work.

I find it highly disturbing that a software engineer of Jamie's caliber would give up on upgrading software. Jamie lives and breathes Linux. It is his platform of choice. If he throws in the towel on Linux upgrades, then what possible hope do us mere mortals have?

Revisiting "Keyboard vs. The Mouse, pt 1"

You may know Bruce Tognazzini from his days as Apple Computer employee #66, or perhaps his classic books Tog on Interface and Tog on Software Design. He's still quite relevant today; his list of the ten most persistent UI bugs is an excellent reminder that many of the biggest computer interface problems are still essentially unsolved.

But what I want to talk about today is one particular Ask Tog column from August 1989, which was republished as Chapter 6 of his book Tog on Interface. It's titled Keyboard vs. The Mouse, pt 1. It contains this quote:

We've done a cool $50 million of R & D on the Apple Human Interface. We discovered, among other things, two pertinent facts:

• Test subjects consistently report that keyboarding is faster than mousing.
• The stopwatch consistently proves mousing is faster than keyboarding.

This contradiction between user-experience and reality apparently forms the basis for many user/developers' belief that the keyboard is faster.

Let's assume that we're typing some text into a document of some kind, and we wish to save the document we're working on. (I could argue that the user should never have to explicitly save anything, but humor me.) If it seems ridiculous that the mouse method:

1. Take your right hand off the keyboard
2. Place your right hand on the mouse
3. Mouse over to the File menu
4. Click File
5. Click Save
6. Place your right hand back on the keyboard

Could be measurably faster than the keyboard method:

1. Use your left hand to press Control+S

I assure you that you are not alone. Please defer all your righteous indignation for just a moment.

First, understand that this is a very old quote. In 1989, the current desktop operating systems were Windows 2.0, DOS 4.0, and Mac System 6. The "people new to the mouse" Tog refers to were keyboard holdouts, convinced that the mouse was nothing more than a giant, gimmicky waste of time. I don't think you'll find many users like that around today. Furthermore, I doubt Tog envisioned today's world of main menus with literally thousands of commands, a menu tree -- no, a menu forest -- so dense that search or some other alternate UI metaphor is the only rational way to navigate it all.

This ancient quote is invariably trotted out during any discussion of keyboard shortcuts, and always for the wrong reasons. The first paragraph is all most people read, so they misunderstand the full meaning. It helps to read the entire column. Tog explains keyboard shortcuts the way modern users understand them just a few short paragraphs later:

And, in fact, I find myself on the opposite side in at least one instance, namely editing. By using Command X, C, and V, the user can select with one hand and act with the other. Two-handed input. Two-handed input can result in solid productivity gains (Buxton 1986).

I don't think anyone would argue that learning keyboard shortcuts is faster than using the mouse to navigate and learn a program. Clearly it isn't -- it's quite painful, as anyone who has ever been stranded at a Unix command prompt can probably tell you.

However, as Tog himself notes, when the keyboard shortcut is already memorized and well understood, it's a clear productivity win.

I've long been an advocate of two-fisted computing -- using both your keyboard and your mouse to the fullest. That's what keyboard shortcuts are to me. I'm not sure why this always has to be spun as a cage match between the keyboard and the mouse. Keyboard shortcuts don't replace my mousing; they complement it.

In my experience, there's absolutely no question that judicious use of a few keyboard shortcuts will make you faster and more productive. But you don't have to take it from me. Just Ask Tog.

Just a Little Bit of Software History Repeating

I lived in the Denver area at the time Denver International Airport's completely computer automated baggage system was unveiled in 1994. The troubled development of this system was big local news.

The premise of Denver's plan was as big as the West. The distance from a centralized baggage check-in to the farthest gate - about a mile - dictated expansive new thinking, planners said, and technology would make the new airport a marvel. Travelers who arrived for check-in or stepped off a plane would have their bags whisked across the airport with minimal human intervention. The result would be fewer flight delays, less waiting at luggage carousels and big savings in airline labor costs.

Tours that preceded the system's debut led invariably to an airport basement where 26 miles of track, loaded with thousands of small gray carts, sped bags up and down inclines as conveyor belts minutely timed by the computer deposited each bag in its cart at just the right moment.

The baggage system was an abject failure. It had huge problems on opening day, and almost immediately had to be superseded by manual procedures. Things never improved much from there. Denver's automated baggage handling system was scrapped completely by 2005, in favor of traditional manual handling and barcode scanning procedures.

"It wasn't the technology per se, it was a misplaced faith in it," said Richard de Neufville, a professor of civil and environmental engineering and engineering systems at the Massachusetts Institute of Technology. Professor de Neufville said the builders had imagined that their creation would work well even at the busiest boundaries of its capacity. That left no room for the errors and inefficiencies that are inevitable in a complex enterprise.

"The main culprit was hubris," he said.

I was surprised, then, to read essentially the same scenario play out 10 years later in England at Heathrow Airport's new Terminal 5.

Many things did not go according to plan at T5, but at the core of the fiasco is baggage. This is supposed to be a state-of-the-art system, the biggest in Europe with 10 miles of conveyor belts controlled by 140 computers and designed to process 12,000 bags per hour at up to 23 mph. But it had never been tested before in a live terminal. On T5's D-Day many aspects -- human and technological -- simply did not work.

BA say they are confident that a few days bedding-down will sort out the problems. However, T5 is not yet at full capacity, with another 70 long-haul destinations due to move there on 30 April.

I sincerely hope BA can escape from Gilligan's Island, unlike so many hardy travellers before them. Otherwise, all we'll end up with is another sad entry in the long, dismal history of software project failure.

What Should The Middle Mouse Button Mean?

Despite Apple's historical insistence that the computer mouse should only have one button-- which led to the highly unfortunate convention of double-clicking-- most mice have more than one button today. In his classic book The Humane Interface, Jef Raskin revisits the earliest days of his involvement with the Mac project and realizes that the single button mouse was a mistake. Mice were meant to have multiple buttons.

What I did not see at the time is that multiple buttons on a mouse can work well if the buttons are labeled. If the Macintosh mouse had had multiple buttons, if the buttons had been permanently labeled, and if they had only been used for their intended function, a multiple mouse button might have been a better choice. A better mouse might have two buttons, marked Select and Activate, on top and on the side, a button activated by a squeezing action of the thumb. This last button would be marked Grab. Some mice at present have a scroll wheel on top that is used primarily for scrolling. Better still would be a small trackball in that location. The mouse would control the position of the cursor; the trackball could be used, for example, to manipulate objects or to make selections from menus that float with the cursor.

Doug Engelbart, the inventor of the mouse, also thinks that mice should have multiple buttons:

[Doug Engelbart] believes a mouse should have many buttons ... the only reason his original mouse design didn't have more than three was because they didn't have the technology at the time to make that possible.

Apple didn't ship a multiple button mouse until the Mighty Mouse was released in August 2005. It has four effective buttons, and even sports the trackball that Jef Raskin imagined in his book five years earlier. However, I've read a lot of complaints about the Mighty Mouse, most of which stem from the substitution of actual buttons with touch-sensitive surfaces.

I've used two-button mice as far back as I can remember on the PC. The meaning of the first two mouse buttons are very well defined in every graphical user interface by now:

But modern mice actually have at least three buttons. Where's the third button? Right under your mouse wheel.

Mouse wheels have been commonly available since 1996. In all those years, all those millions of mice shipped, no standard convention has emerged for what it means to press the middle mouse button.

Over the last two or three years, middle click has become strongly associated with tabbed user interfaces, at least in popular web browsers. Middle-clicking over a link opens it in a new tab; middle-clicking the tab itself closes that tab. This is happening in enough applications now that I think it's fair to call opening and closing tabs with the middle button an emerging convention. Still, it's a fairly loose convention, and the behavior is only defined for links and tabs respectively, and only in certain applications. What happens the rest of the time when you middle-click?

Another odd middle-click behavior that's defined in both Internet Explorer and Firefox is the modal "autoscroll mode". Middle click once on the page to activate this mode. Notice that the cursor changes. You can now use the mouse to determine the rate of scrolling. Middle-clicking again releases this mode and reverts to the normal mouse cursor.

I personally hate this behavior. I prefer to scroll explicitly with the wheel, and I often trigger this unwanted "mode" when I've slightly missed middle-clicking on a link. It can be turned off in the advanced options of Firefox but I can find no way to turn it off in Internet Explorer.

In the UNIX and X Windows world, the middle button has also meant paste since way, way back in the 1980s. I can't find any evidence of this behavior on Windows or the Mac, however. Pasting into text areas wouldn't necessarily conflict with the tab behavior, but it's an odd hodgepodge of behaviors to attach to a single button.

I hope over the next few years Microsoft and Apple can decide on a set of standard middle mouse button behaviors. It's frustrating to me that millions and millions of mice have shipped with this button, and yet it's a total crapshoot what will happen when you press the middle mouse button in any given application under any operating system. If the first and second mouse buttons have standard, well-defined meanings today-- why can't the third button, too?

I {entity} Unicode

These are available as bumper stickers and t-shirts:

Here's my rhetorical question to you: why is this funny?

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
• There Ain't No Such Thing as Plain Text
• On the Goodness of Unicode
• Wikipedia article on Unicode

Here's hoping the next programmer who sees this is laughing, too-- and for the right reasons.

Revisiting The Facts and Fallacies of Software Engineering

I like to re-read my favorite books every few years, so I brought Robert Glass' seminal Facts and Fallacies of Software Engineering with me on my most recent trip. I thought it was a decent, but imperfect read when I originally bought it in 2004. As I scanned through the introduction and table of contents, I realized that I've written about almost everything in this book by now.

I'm not sure I gave Facts and Fallacies its due on my first read.

Simply reciting the various facts and fallacies feels like a zen koan to software engineering. Even without any of the background discussion and explanation in the book, it's therapeutic to ponder the brief one sentence summaries presented in the table of contents. As you read these, what comes to mind, based on your experience?

People

1. The most important factor in software work is the quality of the programmers.
2. The best programmers are up to 28 times better than the worst programmers.
3. Adding people to a late project makes it later.
4. The working environment has a profound impact on productivity and quality.

Tools and Techniques

5. Hype (about tools and technology) is a plague on the house of software.
6. New tools and techniques cause an initial loss of productivity / quality.
7. Software developers talk a lot about tools, but seldom use them.

Estimation

8. One of the two most common causes of runaway projects is poor estimation.
9. Software estimation usually occurs at the wrong time.
10. Software estimation is usually done by the wrong people.
11. Software estimates are rarely corrected as the project proceeds.
12. It is not surprising that software estimates are bad. But we live and die by them anyway!
13. There is a disconnect between software management and their programmers.
14. The answer to a feasability study is almost always "yes".

Reuse

15. Reuse-in-the-small is a solved problem.
16. Reuse-in-the-large remains a mostly unsolved problem.
17. Reuse-in-the-large works best in families of related systems.
18. Reuseable components are three times as hard to build and should be tried out in three different settings.
19. Modification of reused code is particularly error-prone.
20. Design pattern reuse is one solution to the problems of code reuse.

Requirements

23. One of the two most common causes of runaway projects is unstable requirements.
24. Requirements errors are the most expensive to fix during production.
25. Missing requirements are the hardest requirements errors to correct.

Design

26. Explicit requirements 'explode' as implicit requirements for a solution evolve.
27. There is seldom one best design solution to a software problem.
28. Design is a complex, iterative process. Initial design solutions are usually wrong and certainly not optimal.

Coding

29. Designer 'primitives' rarely match programmer 'primitives'.
30. COBOL is a very bad language, but all the others are so much worse.

Error removal

31. Error removal is the most time-consuming phase of the lifecycle.

Testing

32. Software is usually tested at best to the 55 to 60 percent coverage level.
33. 100 percent test coverage is still far from enough.
34. Test tools are essential, but rarely used.
35. Test automation rarely is. Most testing activities cannot be automated.
36. Programmer-created, built-in debug code is an important supplement to testing tools.

Reviews and Inspections

37. Rigorous inspections can remove up to 90 percent of errors before the first test case is run.
38. Rigorous inspections should not replace testing.
39. Post-delivery reviews, postmortems, and retrospectives are important and seldom performed.
40. Reviews are both technical and sociological, and both factors must be accommodated.

Maintenance

41. Maintenance typically consumes 40 to 80 percent of software costs. It is probably the most important software lifecycle phase.
42. Enhancements represent roughly 60 percent of maintenance costs.
43. Maintenance is a solution-- not a problem.
44. Understanding the existing product is the most difficult maintenance task.
45. Better methods lead to more maintenance, not less.

Quality

46. Quality is a collection of attributes.
47. Quality is not user satisfaction, meeting requirements, achieving cost and schedule, or reliability.

Reliability

48. There are errors that most programmers tend to make.
49. Errors tend to cluster.
50. There is no single best approach to software error removal.
51. Residual errors will always persist. The goal should be to minimize or eliminate severe errors.

Efficiency

52. Efficiency stems more from good design than good coding.
53. High-order language code can be about 90 percent as efficient as comparable assembler code.
54. There are tradeoffs between optimizing for time and optimizing for space.

Research

55. Many researchers advocate rather than investigate.

I had forgotten how much ground the book covers; it's a perfect springboard to all the essential topics in software engineering.

I've posted on almost every one of these facts in the intervening four years since I originally read them. As I delved into the table of contents presented above, I could barely contain myself. I remembered and mentally checked each post off the list as I went: check, check, check. I've been accused of gratuitous self-linking in the past, so I won't clutter up the rules with dozens of links to my old posts on these topics. If you're interested, you can find it. That's sort of the point.

If those are the fifty-five facts, then these are the ten fallacies presented at the end. Fallacies have the ring of truth, but upon closer inspection, turn out to be problematic when applied to a real live software project.

1. You can't manage what you can't measure.
2. You can manage quality into a software product.
3. Programming can and should be egoless.
4. Tools and techniques: one size fits all.
5. Software needs more methodologies.
6. To estimate cost and schedule, first estimate lines of code.
7. Random test input is a good way to optimize testing.
8. "Given enough eyeballs, all bugs are shallow".
9. The way to preduct future maintenance costs and to make product replacement decisions is to look at past cost data.
10. You teach people how to program by showing them how to write programs.

If you're curious about the rationale behind these facts and fallacies, that's entirely the reason the book exists: to remind us to question what we're doing. We should be thinking about our craft every day, in some small way, on our own software projects. That's how we collectively advance software engineering-- by building our shared memory and history in the field. As Mr. Glass states in the introduction:

In presenting these facts, I am also indentifying problems in the field. It is not my intention to present solutions to these problems. This is a what-is book, not a how-to book. That's important to me. I want to bring these facts into the open, where they can be freely discussed, and we can act on them to make progress.

I encourage you to pick up a copy of the full book for a deeper exploration. I do believe there's a rich learning experience-- or a rich remembering experience-- here for those of you who choose to read on.

Is Eeyore Designing Your Software?

This classic Eric Lippert post describes, in excruciating, painful detail, exactly how much work it takes to add a single ChangeLightBulbWindowHandleEx function to a codebase at Microsoft:

One dev to spend five minutes implementing ChangeLightBulbWindowHandleEx.

One program manager to write the specification.

One localization expert to review the specification for localizability issues.

One usability expert to review the specification for accessibility and usability issues.

At least one dev, tester and PM to brainstorm security vulnerabilities.

One PM to add the security model to the specification.

One tester to write the test plan.

One test lead to update the test schedule.

One tester to write the test cases and add them to the nightly automation.

Three or four testers to participate in an ad hoc bug bash.

One technical writer to write the documentation.

One technical reviewer to proofread the documentation.

One copy editor to proofread the documentation.

One documentation manager to integrate the new documentation into the existing body of text, update tables of contents, indexes, etc.

Twenty-five translators to translate the documentation and error messages into all the languages supported by Windows.The managers for the translators live in Ireland (European languages) and Japan (Asian languages), which are both severely time-shifted from Redmond, so dealing with them can be a fairly complex logistical problem.

A team of senior managers to coordinate all these people, write the cheques, and justify the costs to their Vice President.

I think sometimes programmers forget how much work it is to create software at large companies. What may seem like a no-brainer five line code change to us on the outside is perhaps five man-weeks of work once you factor in all the required process overhead. We're picking on Microsoft here, but this is by no means limited to Microsoft; it's a simple function of scale and audience for all commercial software.

So then, the obvious question: who does all those things for non-commercial, open source software? The answer, per a Raymond Chen comment on the same post, is "nobody":

Who develops the test plans for open source software? Who updates the screenshots in the user's guide and online help? And who translates the documentation into Polish and Turkish? Who verifies that the feature doesn't violate the Americans with Disabilities Act or German privacy laws? Back when I worked on Linux, the answer was "Nobody. There is no test plan, there is no printed user's guide, what little documentation there is exists only in English, and nobody cares about complying with the ADA or German privacy laws." Maybe things have changed since then.

Here's my honest question: does open source software need all that process to be successful? Isn't the radical lack of process baggage in open source software development not a weakness, but in fact an evolutionary advantage? What open source software lacks in formal process it makes up ten times over in ubiquity and community. In other words, if the Elbonians feel so strongly about localization, they can take that effort on themselves. Meanwhile, the developers have more time to implement features that delight the largest base of customers, instead of plowing through mountains of process for every miniscule five line code change.

Are large commercial software companies crippled by their own process?

If you openly reward and promote people for killing work by bemoaning the risk and the testing cost and localization impact of each feature and interrogating a design change request as if it were Dan Brown shackled in-front of a wild-eyed, hot-poker wielding Pope, well, everyone is going to grab pitchforks and jump on that "No can do! No can ship!" bandwagon.

It makes me think of how many feature meetings I've had and what a small percent of those features have actually ever shipped. Not that every feature is a good idea, but it's damn near wake-worthy sometimes for a feature to actually get out into shipping bits. Que Eeyore: "Oh no. Now we have to support it. I suppose a hotfix request will come in any moment now..."

All too often, it really does feel like Microsoft's software was designed by Eeyore.

In this case, the bird represents features that delight customers.

The Sierra Network II

You may remember Sierra's ImagiNation network from the earliest days of dial-up networking:

The ImagiNation Network (INN), aka The Sierra Network (TSN), was the first online multiplayer gaming system. Developed by Sierra On-Line in 1989, and first available to the public in 1991, the ImagiNation Network was a unique online gaming network that gave subscribers from all over the United States of America a place where they could "play games, make friends and have fun". With a wide variety of games including RPGs, WWI aeroplane simulations, live trivia, and card and board games, almost every user could find something enjoyable to play. INN also featured an electronic post office, many bulletin boards, chat rooms, and the company boasted of having "more than 200 groups, clubs and special events online."

I had an account on The Sierra Network for a while. The graphics were incredible for that era, at least compared to the text-only BBS games that passed for online multiplayer gaming at the time. Still, it wasn't quite my cup of tea, so I didn't last long there. I finally achieved online multiplayer satisfaction a few years later with Doom, Dwango, and Kali.

The Sierra Network is an interesting bit of computer history trivia at best. But it's particularly relevant when you compare it to the recently launched Mytopia gaming service. Mytopia allows you to play common internet games (think hearts, sudoku, chess, etcetera) across several popular walled garden social networking sites including MySpace and Facebook.

The resemblance, indeed, is astonishing. It has to be some sign of the coming internet apocalypse when a startup has essentially rebuilt The Sierra Network in Web 2.0 fashion.

(via rei on QuarterToThree)

Paul Graham's Participatory Narcissism

I have tremendous respect for Paul Graham. His essays-- repackaged in the book Hackers and Painters-- are among the best writing I've found on software engineering. Not all of them are so great, of course, but the majority are well worth your time. That's more than I can say for 99.9-infinitely-repeating-percent of the content on the web. He's certainly a better and more authoritative writer than I.

But lately I've begun to wonder whether Mr. Graham, like Joel Spolsky before him, has devolved into self-absorption and irrelevance. Consider his latest essay, You Weren't Meant to Have a Boss, which opens with this distasteful anecdote:

A few days ago I was sitting in a cafe in Palo Alto and a group of programmers came in on some kind of scavenger hunt. It was obviously one of those corporate "team-building" exercises.

They looked familiar. I spend nearly all my time working with programmers in their twenties and early thirties. But something seemed wrong about these. There was something missing.

And yet the company they worked for is considered a good one, and from what I overheard of their conversation, they seemed smart enough. In fact, they seemed to be from one of the more prestigious groups within the company. So why did it seem there was something odd about them?

The guys on the scavenger hunt looked like the programmers I was used to, but they were employees instead of founders. And it was startling how different they seemed.

So what, you may say. So I happen to know a subset of programmers who are especially ambitious. Of course less ambitious people will seem different. But the difference between the programmers I saw in the cafe and the ones I was used to wasn't just a difference of degree. Something seemed wrong.

I think it's not so much that there's something special about founders as that there's something missing in the lives of employees. I think startup founders, though statistically outliers, are actually living in a way that's more natural for humans.

I was in Africa last year and saw a lot of animals in the wild that I'd only seen in zoos before. It was remarkable how different they seemed. Particularly lions. Lions in the wild seem about ten times more alive. They're like different animals. And seeing those guys on their scavenger hunt was like seeing lions in a zoo after spending several years watching them in the wild.

I'm not sure why Mr. Graham felt the need to draw this incredibly condescending parallel with company employees and caged animals in the zoo.

I've actually taken Mr. Graham's advice. I recently quit my job to blog and participate in a micro startup. Even though I'm now one of the anointed founders in Mr. Graham's book, I still found this comparison retroactively offensive to all those years I worked as an employee for various companies and had perfectly enriching, rewarding-- dare I say even enjoyable-- experiences. Or at least as happy as a caged animal in a zoo can ever be, I suppose.

Mr. Graham's essay does contain some fair points, if you can suppress your gag reflex long enough to get to them. If you don't have time to read it, lex99 posted this succinct summary that I thought captured its flavor perfectly:

I work with young startup founders in their twenties. They're geniuses, and play by their own rules. Oh... you haven't founded a company? You suck.

Small businesses are the backbone of the American economy. And Mr. Graham is absolutely right to encourage young people to take risks early in life, to join small business startups with potentially limitless upside while they have nothing to lose -- no children, no mortgage, no significant other. I believe in this so strongly I included it as a slide in my presentation to graduating Canadian computer science students.

Indeed, you should take insane career risks while you're young.

And there are lots of large corporate soul-sucking programming jobs that are, quite literally, Dilbert cartoons brought to life.

The problem with this particular essay is the way Mr. Graham implies the only path to true happiness as a young programmer lies in founding a startup. If you aren't a founder, or one of the first 10 employees, then, well.. enjoy your life at the zoo. We'll be sure to visit when we aren't busy loping free on the plains, working the way people were meant to. I'm not paraphrasing here; he actually wrote that: working the way people were meant to. The sense of disdain, the dismissiveness, is nearly palpable.

He acknowledges that his perspective is warped because "nearly all the programmers [he knows] are startup founders." Therein lies the problem. These essays are no longer about software engineering; they're about Paul Graham. They've become participatory narcissism:

After a while, you begin to notice that all the essays are an elaborate set of mirrors set up to reflect different facets of the author, in a big distributed act of participatory narcissism.

Naturally, every young software programmer worth a damn forms a startup. Because that's what Mr. Graham's company, Y Combinator, does. They fund startups with young software programmers. He projects his reality outward, reflecting it against the rest of us so brightly and so strongly that we're temporarily blinded. We stop seeing our own reality and trade it for his, in a form of participatory narcissism-- we believe in the one true path to success, exactly the way Mr. Graham has laid it before us. Traditional employment? That's for suckers. Real go-getters start their own companies.

On the whole, I think I preferred Paul Graham's essays when they were more about software engineering and less about Paul Graham.

Update: Paul Graham posted two essays that partially respond to this post: You Weren't Meant to Have a Boss: The Cliffs Notes and How to Disagree. The latter is, as far as I can tell, a sort of EULA for disagreeing with Paul Graham. Based on the conversation this post initiated, I attended a Y Combinator dinner and got to meet Mr. Graham in person. That is, to me, the point of posts like this -- some initial disagreement ultimately leading to deeper, more satisfying communication. A net positive all around.

The First Rule of Programming: It's Always Your Fault

You know the feeling. It's happened to all of us at some point: you've pored over the code a dozen times and still can't find a problem with it. But there's some bug or error you can't seem to get rid of. There just has to be something wrong with the machine you're coding on, with the operating system you're running under, with the tools and libraries you're using. There just has to be!

No matter how desperate you get, don't choose that path. Down that path lies voodoo computing and programming by coincidence. In short, madness.

It's frustrating to repeatedly bang your head against difficult, obscure bugs, but don't let desperation lead you astray. An essential part of being a humble programmer is realizing that whenever there's a problem with the code you've written, it's always your fault. This is aptly summarized in The Pragmatic Programmer as "Select Isn't Broken":

In most projects, the code you are debugging may be a mixture of application code written by you and others on your project team, third-party products (database, connectivity, graphical libraries, specialized communications or algorithms, and so on) and the platform environment (operating system, system libraries, and compilers).

It is possible that a bug exists in the OS, the compiler, or a third-party product-- but this should not be your first thought. It is much more likely that the bug exists in the application code under development. It is generally more profitable to assume that the application code is incorrectly calling into a library than to assume that the library itself is broken. Even if the problem does lie with a third party, you'll still have to eliminate your code before submitting the bug report.

We worked on a project where a senior engineer was convinced that the select system call was broken on Solaris. No amount of persuasion or logic could change his mind (the fact that every other networking application on the box worked fine was irrelevant). He spent weeks writing workarounds, which, for some odd reason, didn't seem to fix the problem. When finally forced to sit down and read the documentation on select, he discovered the problem and corrected it in a matter of minutes. We now use the phrase "select is broken" as a gentle reminder whenever one of us starts blaming the system for a fault that is likely to be our own.

The flip side of code ownership is code responsibility. No matter what the problem is with your software-- maybe it's not even your code in the first place-- always assume the problem is in your code and act accordingly. If you're going to subject the world to your software, take full responsibility for its failures. Even if, technically speaking, you don't have to. That's how you earn respect and credibility. You certainly don't earn respect or credibility by endlessly pawning off errors and problems on other people, other companies, other sources.

Statistically, you understand, it is incredibly rare for any bugs or errors in your software not to be your fault. In Code Complete, Steve McConnell cited two studies that proved it:

A pair of studies performed [in 1973 and 1984] found that, of total errors reported, roughly 95% are caused by programmers, 2% by systems software (the compiler and the operating system), 2% by some other software, and 1% by the hardware. Systems software and development tools are used by many more people today than they were in the 1970s and 1980s, and so my best guess is that, today, an even higher percentage of errors are the programmers' fault.

Whatever the problem with your software is, take ownership. Start with your code, and investigate further and further outward until you have definitive evidence of where the problem lies. If the problem lies in some other bit of code that you don't control, you'll not only have learned essential troubleshooting and diagnostic skills, you'll also have an audit trail of evidence to back up your claims, too. This is certainly a lot more work than shrugging your shoulders and pointing your finger at the OS, the tools, or the framework-- but it also engenders a sense of trust and respect you're unlikely to achieve through fingerpointing and evasion.

If you truly aspire to being a humble programmer, you should have no qualms about saying "hey, this is my fault-- and I'll get to the bottom of it."