Every User Lies

Heidi Adkisson notes that features sell products, but the people buying those products often don't use the very features they bought the product for in the first place.

A few years ago I did an extensive in-home study observing use of a particular computer hardware peripheral. Most people had high-end models with many features. But in my observation of use, only one "power user" went beyond using anything but the core, basic features. These people had paid a premium for features they didn't use. However, when describing their purchase experience, it was clear they aspired to using these features and sincerely intended to. But, once the product was out of the box, the paradox of the active user took over. They never invested even the smallest amount of time to go beyond the basics (even though the extra features could have saved them time).

In my experience it's people's aspiration for the experience that drives purchasing decisions. Ultimately, their aspirations may be different than the reality.

It's interesting that Heidi used the hedge phrase "may be different" when her own study data showed that users' aspirations and reality were almost always different. Maybe she aspires to live in a world where aspirations and reality aren't so wildly divergent. I don't blame her. It'd be nice.

That disparity is why it's so important to observe how users actually behave versus the way they tell you they behave. People who do this professionally are called "economists". Observation is a powerful skill, and so is learning to disregard what people tell you in favor of judging them by their actions. No actions are more carefully considered than those that result in money flowing out of your pocket. That's why you owe it to yourself to read books like Freakonomics, and maybe even The Economist magazine. It's also why the Freakonomics blog should be a part of your regular reading routine if it isn't already.

People lie not because they're all evil liars (although a few inevitably will be), but because they're usually lying to themselves in some way. Some lies are useful. Small social "white" lies grease the skids of social reality. Penetrating this veil of lies and intentions is one of the central themes of the excellent television show House, M.D. :

The show plays up subtle connections between the House character and Sherlock Holmes, which is appropriate, because it's very much a detective show at heart. The character Gregory House, as played by the brilliant Hugh Laurie, is fond of stating "Everybody lies." Parsing through all the irrational human behavior, and the inevitable lies-- white or otherwise-- makes for a gripping detective story indeed when lives are at stake.

Heidi referenced the Paradox of the Active User (pdf), which has been around as a concept since 1987. I highly recommend reading the original paper, but if you don't have time, Jakob Nielsen summarizes:

Users never read manuals but start using the software immediately. They are motivated to get started and to get their immediate task done: they don't care about the system as such and don't want to spend time up front on getting established, set up, or going through learning packages.

The "paradox of the active user" is a paradox because users would save time in the long term by learning more about the system. But that's not how people behave in the real world, so we cannot allow engineers to build products for an idealized rational user when real humans are irrational. We must design for the way users actually behave.

There are a bunch of ways to restate the paradox of the active user. Cooper calls it perpetual intermediacy. I think the easiest way to explain it is this: every user lies. Instead of asking users if they love your software-- of course they love your software, it'd be rude to tell the people responsible just how mind-numbingly awful it really is-- do what Gregory House does. Observe whether or not they use the software, and how they use it. Rely on that behavioral data to design your software, and not the lies of your users, however well intentioned they may be.

Is Worse Really Better?

You may think of Steve Martin as a stereotypical family friendly comedian today-- the center of saccharine movies like Parenthood and Father of the Bride. But it wasn't always this way. Steve hit his stride in the early 80's. At that time, I don't think there were any popular comedians exploring the ragged edge of humor in quite the same way Steve Martin was. I'll forever remember finding a copy of his book Cruel Shoes as an impressionable teenager. It's a collection of very strange short stories. At my tender young age, I had certainly never read anything like it. It's hard to explain. Read for yourself. Here's the complete text of the eponymous Cruel Shoes short story:

Anna knew she had to have some new shoes today, and Carlo had helped her try on every pair in the store. Carlo spoke wearily, "Well, that's every pair of shoes in the place."

"Oh, you must have one more pair ..."

"No, not one more pair...Well, we have the cruel shoes, but no one would want..."

Anna interrupted, "Oh yes, let me see the cruel shoes!"

Carlo looked incredulous. "No, Anna, you don't understand, you see, the cruel shoes are..."

"Get them!"

Carlo disappeared into the back room for a moment, then returned with an ordinary shoebox. He opened the lid and removed a hideous pair of black and white pumps. But these were not an ordinary pair of black and white pumps; both were left feet, one had a right angle turn with seperate compartments that pointed the toes in impossible directions. The other shoe was six inches long and was curved inward like a rocking chair with a vise and razor blades to hold the foot in place. Carlo spoke hesitantly, "...Now you see why...they're not fit for humans..."

"Put them on me."

"But..."

"Put them on me!"

Carlo knew all arguments were useless. He knelt down before her and forced the feet into the shoes.

The screams were incredible.

Anna crawled to the mirror and held her bloody feet up where she could see.

"I like them."

She paid Carlo and crawled out of the store into the street.

Later that day, Carlo was overheard saying to a new customer, "Well, that's every shoe in the place. Unless, of course, you'd like to try the cruel shoes."

Funny, yes, but also disturbing-- you feel vaguely uncomfortable while laughing at Cruel Shoes. That awkward feeling is what makes Steve's humor so subversive and thought provoking. I previously cited my lifelong fascination with the subversive humor of Mad Magazine, and I'd hold Steve Martin's brand of strange, experimental comedy right up there next to it. I distinctly remember seeing The Jerk in 1979. I could only make sense of about half of it at the time, but the half I did understand blew my young mind. I've been slowly deciphering the genius of this Steve Martin movie ever since.

In a recent interview with Smithsonian Magazine, Steve explains some of the philosophy behind his unique comedy. It's a fantastic article, but one particular passage stood out:

At the end of my closing-night show at the Troubadour, I stood onstage and took out five bananas. I peeled them, put one on my head, one in each pocket and squeezed one in each hand. Then I read the last line of my latest bad review: "Sharing the bill with Poco this week is comedian Steve Martin ... his 25-minute routine failed to establish any comic identity that would make the audience remember him or the material." Then I walked off the stage.

The consistent work enhanced my act. I learned a lesson: it was easy to be great. Every entertainer has a night when everything is clicking. These nights are accidental and statistical: like lucky cards in poker, you can count on them occurring over time. What was hard was to be good, consistently good, night after night, no matter what the circumstances.

Steve's insistence that greatness isn't something you can count on, or even something you should strive for, resonates deeply for me. Greatness is far too difficult, too abstract, too daunting. Being good-- consistently good-- is the real goal, and that takes hard work and discipline. Being good-- that's something concrete you can roll up your sleeves and accomplish. Forget greatness. Can we even define what greatness truly is? Like Steve Martin, you become great through applying yourself at being reliably good, night after night, venue after venue, time after time.

Voltaire originally said better is the enemy of good; at the risk of creating another snowclone, Steve's advice is essentially that great is the enemy of good. It's not exactly a message about software development, but it strongly reminds me of worse is better, at least from where I'm sitting. There's a fascinating history behind this classic essay that I glossed over in my original post on the topic. After doing a bit more research, I found that Richard Gabriel wrote a detailed article explaining the rich history of worse is better:

The concept known as "worse is better" holds that in software making (and perhaps in other arenas as well) it is better to start with a minimal creation and grow it as needed. Christopher Alexander might call this "piecemeal growth." This is the story of the evolution of that concept.

From 1984 until 1994 I had a Lisp company called "Lucid, Inc." In 1989 it was clear that the Lisp business was not going well, partly because the AI companies were floundering and partly because those AI companies were starting to blame Lisp and its implementations for the failures of AI. One day in Spring 1989, I was sitting out on the Lucid porch with some of the hackers, and someone asked me why I thought people believed C and Unix were better than Lisp. I jokingly answered, "because, well, worse is better." We laughed over it for a while as I tried to make up an argument for why something clearly lousy could be good.

The idea being, of course, that enough goodness slowly accreted over time usually trumps any epic (and usually ill-fated) plans to create any brand new great thing. I completely agree, and I think history bears this lesson out a hundredfold. But even the original author, Richard Gabriel, can't decide if worse really is better. He's written a slew of pro and con articles over the years, vacillating back and forth, playing Devil's Advocate to his own position, representing both the "worse" and "better" sides in equal measure:

• Lisp: Good News, Bad News, How to Win Big (original "Worse is Better" talk, 1991)
• Worse Is Better is Worse (pdf, 1991)
• Is Worse Really Better? (pdf, 1992)
• Models of Software Acceptance: How Winners Win (pdf presentation, 1995)
• Money Through Innovation Reconsidered (pdf, 1995)
• Back to the Future: Is Worse (Still) Better? (pdf, 2000)
• Back to the Future: Worse (Still) is Better! (pdf, 2000)

Richard's final statement on the matter is a bit of a cop out: decide for yourselves. I'm not sure if worse is better or not. Personally, I'm inclined to follow Steve Martin's advice here: strive to be consistently good, and the greatness takes care of itself.

What's Your Backup Strategy?

Jamie Zawinski's public service backup announcement starts off with a bang:

Option 1: Learn not to care about your data. Don't save any old email, use a film camera, and only listen to physical CDs and not MP3s. If you have no possessions, you have nothing to lose.

This is obviously meant as satire, but it's disturbingly close to reality for me. I suppose everything in my life that's worth capturing... well, let me put it this way: you're reading it. When I said make it public, I really meant it. Still, I'm fairly sure Jamie was kidding, and while Google may be a great service, it's only a so-so backup mechanism. Let's proceed to option 2, which goes something like this:

1) You have a computer. It came with a hard drive in it. Go buy two more drives of the same size or larger. If the drive in your computer is SATA2, get SATA2. If it's a 2.5" laptop drive, get two of those. Brand doesn't matter, but physical measurements and connectors should match.

2) Get external enclosures for both of them. The enclosures are under $30.

3) Put one of these drives in its enclosure on your desk. Name it something clever like "Backup". If you are using a Mac, the command you use to back up is this:

sudo rsync -vaxE --delete --ignore-errors / /Volumes/Backup/

If you're using Linux, it's something a lot like that. If you're using Windows, go f*ck yourself.

Yeah! Take that, Windows users! Hey, wait a second. I use Windows. Did I mention that Jamie is a funny guy? Moving on.

I've long been a fan of inexpensive hard drive enclosures. Jamie's advice confirms my long held opinion that multiple hard drives are the most effective and easy backup process you'll ever find. The rsync command is more than a simple copy; it actually does a block-by-block comparison, only copying the differences. So instead of backing up the entire contents of your hard drive (again), you only back up the parts that changed since your last backup. This is commonly known as incremental backup.

Incremental backups only have value if you're doing them regularly, so it's only natural to schedule this as a recurring task.

4) If you have a desktop computer, have this happen every morning at 5AM by creating a temporary text file containing this line:

0 5 * * * rsync -vaxE --delete --ignore-errors / /Volumes/Backup/

and then doing sudo crontab -u root that-file

If you have a laptop, do that before you go to bed. Really. Every night when you plug your laptop in to charge.

5) If you're on a Mac, that backup drive will be bootable. That means that when (WHEN) your internal drive scorches itself, you can just take your backup drive and put it in your computer and go. This is nice.

6) When (WHEN) your backup drive goes bad, which you will notice because your last backup failed, replace it immediately. This is your number one priority. Don't wait until the weekend when you have time, do it now, before you so much as touch your computer again. Do it before goddamned breakfast. The universe tends toward maximum irony. Don't push it.

7) That third drive? Do a backup onto it the same way, then take that to your office and lock it in a desk. Every few months, bring it home, do a backup, and immediately take it away again. This is your "my house burned down" backup.

What I like about Jamie's approach is that it's totally KISS, yet it touches all the cornerstones of a solid backup strategy:

1. Pick a simple backup strategy you can live with.
2. Make incremental backups a part of your daily routine.
3. Include an off-site backup in your strategy.

And for the dissenters, although I can't imagine too many with the minimalist backup process Jamie outlined, there's this bon mot:

"OMG, three drives is so expensive! That sounds like a hassle!" Shut up. I know things. You will listen to me. Do it anyway.

I'm not sure Windows users have a direct equivalent of rsync. There is, of course, RoboCopy, and it looks like someone has ported rsync to Windows. But let's face it. I'm a Windows user. When I have a problem, I buy software. That's why, after hearing so many great things about it, I recently purchased a copy of Acronis True Image.

Acronis does a lot of things, but most of all it's drive imaging software, a fancy GUI over the rsync command. With Jamie's recommended two external hard drives in tow, I can use Acronis to create a bootable mirror image of my hard drive. If anything at all goes wrong, I simply swap hard drives, and I'm back in business. I can even create those backup images incrementally and on a schedule. You don't even technically need a second or third hard drive; if you have a large enough primary drive, Acronis will allow you to create a new, hidden partition to store a complete backup image. You can restore these disk images from within Windows proper, during pre-boot, or from bootable USB or optical media. It is very cool, a logical evolution of the more primitive drive imaging products I've used for years.

Of course, as much as I am enamored of it, you don't have to spend thirty bucks on Acronis and even more for two external hard drives to have a decent backup strategy. Lots of people use completely internet based backup services, like Mozy, Carbonite, or JungleDisk, with varying degrees of success. One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

Why Doesn't Anyone Give a Crap About Freedom Zero?

I never quite made the transition from the Apple II series to the Mac. Instead, I migrated from my Apple II to a PC. I always thought the PC ecosystem, although deeply flawed, was more naturally analogous to the eclectic third party hardware and software hacker ecosystem that grew up around the semi-open Apple II hardware platform. This, to me, was the most enduring and beloved quality of the early Apple community. The Mac, in contrast, was underwritten and driven by primarily Apple software running on completely locked down Apple hardware. It's almost first party only-- about as close as you can get to a console platform and still call yourself a computer. I guess you'd say I chose Woz over Jobs. The way Jobs ruthlessly crushed the fledgling clone market in 1997 only reinforced this lesson for me.

So let's be completely clear: when you buy a new Mac, you're buying a giant hardware dongle that allows you to run OS X software.

You know, a dongle:

A dongle is a small hardware device that connects to a computer, often to authenticate a piece of software. When the dongle is not present, the software runs in a restricted mode or refuses to run. Dongles are used by some proprietary vendors as a form of copy prevention or digital rights management because it is much harder to copy the dongle than to copy the software it authenticates. Vendors of software protection dongles (and dongle-controlled software) often use terms such as hardware key, hardware token, or security device in their written literature. In day-to-day use however, the jargon word "dongle" is much more commonly used.

There's nothing harder to copy than an entire MacBook. When the dongle-- or, if you prefer, the "Apple Mac"-- is present, OS X and Apple software runs. It's a remarkably pretty, well-designed machine, to be sure. But let's not kid ourselves: it's also one hell of a dongle.

If the above sounds disapproving in tone, perhaps it is. There's something distasteful to me about dongles, no matter how cool they may be. But it's seductive, too. I wonder if the console model that Jobs is aping isn't some temporary evolutionary dead end, but in fact, the model for all future computing. People buy consoles like the Xbox 360 and Wii because they work with a minimum of fuss. Similarly, people buy Apple hardware because of the perfect synergy between the Apple hardware, OS X, iTunes, iLife, iMovie, iPhoto, and countless other software packages expressly designed to run on a closed hardware platform. "It just works." And why wouldn't it? There are no crude, selfish third parties to screw the experience up behind your back. No oddball hardware, no incompatible drivers, no more software which has to deal with a combinatorial explosion of potential configurations. Choosing to run proprietary software and hardware is just that, a choice. If it's working for consumers, who am I to judge?

I find Apple's brand of hardware lock-in particularly egregious. On the other hand, I run Windows, so I'm subject to my own flavor of self imposed software lock-in. Others have made different choices. In of canaries and coal mines, Mark Pilgrim revisits his choice to abandon Apple's proprietary software model for the world of free software.

18 months later, Apple has sold 4 million crippled phones, billions of crippled songs, and people are predicting that Mac sales are up 40% year over year. And I wouldn't bet against their new movie rental venture either.

So after 18 months, I think we can safely say that no, Cory and I were not "canaries in the coal mine." There are not hordes of fed-up consumers rejecting Apple's vision of cryptographic lock-in. There are not mass graves where people ceremoniously dump their crippled, non-general-purpose computing devices. Outside of Planet Debian and my own personal echo chamber, nobody gives a sh*t about Freedom 0.

You knew this, of course, but I just wanted to let you know that I knew, too.

Maybe I'm a hypocrite. Maybe the issue cuts philosophically deeper than mere dongles. Maybe it's not only about the freedom to run your operating system on whatever hardware you wish, but also the freedom to run whatever software you want for whatever purpose you need, in perpetuity. That's Freedom Zero:

Freedom 0 is the freedom to run the program, for any purpose. WordPress gives me that freedom; Movable Type does not. It never really did, but it was free enough so we all looked the other way, myself included. But Movable Type 3.0 changes the rules, and prices me right out of the market. I do not have the freedom to run the program for any purpose; I only have the limited set of freedoms that Six Apart chooses to bestow upon me, and every new version seems to bestow fewer and fewer freedoms. With Movable Type 2.6, I was allowed to run 11 sites. In 3.0, that right will cost me $535.

WordPress is Free Software. Its rules will never change. In the event that the WordPress community disbands and development stops, a new community can form around the orphaned code. It's happened once already. In the extremely unlikely event that every single contributor (including every contributor to the original b2) agrees to relicense the code under a more restrictive license, I can still fork the current GPL-licensed code and start a new community around it. There is always a path forward. There are no dead ends.

Movable Type is a dead end. In the long run, the utility of all non-Free software approaches zero. All non-Free software is a dead end.

It's compelling rhetoric. As a software developer, there's no denying that open source software is a powerful and transformative force in modern software development.

The console model, and Apple's de-facto first party development model, are about as far as you can get from Mark's freedom zero-- instead, you get zero freedom. You hand the vendor a pile of cash and they allow you to do a handful of specific things with their device, for only so long as they're inclined to do so. It's hardly fair. In fact it's completely unfair; they can legally pull the rug out from under you at any time. But it can still result in some incredibly useful relationships with products that solve very real problems for the user. As Jaron Lanier notes, the iPhone was not a product of freedom zero:

Twenty-five years later, that concern seems to have been justified. Open wisdom-of-crowds software movements have become influential, but they haven't promoted the kind of radical creativity I love most in computer science. If anything, they've been hindrances. Some of the youngest, brightest minds have been trapped in a 1970s intellectual framework because they are hypnotized into accepting old software designs as if they were facts of nature. Linux is a superbly polished copy of an antique, shinier than the original, perhaps, but still defined by it.

Before you write me that angry e-mail, please know I'm not anti-open source. I frequently argue for it in various specific projects. But a politically correct dogma holds that open source is automatically the best path to creativity and innovation, and that claim is not borne out by the facts.

Why are so many of the more sophisticated examples of code in the online world-- like the page-rank algorithms in the top search engines or like Adobe's Flash-- the results of proprietary development? Why did the adored iPhone come out of what many regard as the most closed, tyrannically managed software-development shop on Earth? An honest empiricist must conclude that while the open approach has been able to create lovely, polished copies, it hasn't been so good at creating notable originals. Even though the open-source movement has a stinging countercultural rhetoric, it has in practice been a conservative force.

So I'll ask again, since Mark brought it up: why doesn't anyone give a crap about freedom zero?

What Can You Build in 600 Lines of Code?

Joseph Cooney reminds us that, in January 2005, 37signals went live with a product they built in 579 lines of code:

You read that right, not 60,000 or 600,000 but instead a commercial project written in less than 600 lines of Ruby code. When I first saw this number I was incredulous -- I've written stored procedures that are longer than that. My current project has more lines of configuration than that. I've even written console apps in notepad, and compiled from the command line with more lines than that, because I thought they were so small they didn't need a whole .sln and .proj file, and yet here is 37signals going live with a product that is just 579 lines of Ruby.

As noted in the Rails blog, the original product launch was covered on Ruby language creator Matz' blog in his native Japanese. Surprisingly, the relevant facts are still readable:

Of course, a simple lines of code number isn't the entire story-- they actually built the entire Rails framework first to support building small apps like ta-da list. None of the required Rails framework code, nor any of the the necessary stylesheets, JavaScript, HTML, and so forth, are included in that number. Still, I agree with Joseph: it's an impressive achievement, and it can lead to some interesting thought experiments:

I have a few interesting product ideas from time to time. What is the absolute minimum amount of code I could write that would make those ideas work? If I'm prepared to operate within the constraints of the platform (whatever that is) how much effort would that save me? How many more "interesting ideas" could I turn into working products if I was prepared to follow these constraints? How many more cool/useful things could you build if you promised yourself that each one would only be 600 lines of code?

What can you build in 600 lines of code? Think of it as an exercise in minimalism. Does your preferred language or environment allow you the freedom to create something interesting and useful with that constraint in place?

Getting the Interview Phone Screen Right

The job market for software developers is hot. This is great news for programmers, but it makes the interview process challenging for potential employers. A reader recently wrote me expressing some concern about the interview process:

You mention Vertigo requiring a code sample, then a phone screening, then a hands-on test in the face-to-face. We have a very similar process, but somehow a large percentage of the candidates who make it to the hands-on test are very poor and should have been eliminated at step 1 or 2. The signal to noise ratio is terrible. It costs a great deal to spend so much time doing face-to-face interviews with people who often should not be developers in the first place. I am curious how much light you might be able to shed on the specifics of your requirements on candidates. What part of the process is the most effective in separating the cream, how and why?

It is very expensive to get the phone screen wrong-- a giant waste of time for everyone involved.

The best phone screen article you'll ever find is Steve Yegge's Five Essential Phone-Screen Questions, another gift to us from Steve's stint at Amazon.

Steve starts by noting two critical mistakes that phone screeners should do their best to avoid:

1. Don't let the candidate drive the interview. The interviewer should do most of the talking, guiding the conversation along until they're satisfied the candidate knows the answers to the questions (or has given up).
2. Watch out for one-trick ponies. Candidates who only know one particular language or programming environment, and protest complete ignorance of everything else, are a giant red warning flag.

The point of the phone screen is not for the candidate to drone on about what they've done. The interviewer should push them out of their comfort zone a bit and ask them related questions about things they haven't seen or done before. Ideally, you want to know how this person will react when they face something new, such as your codebase.

In an effort to make life simpler for phone screeners, I've put together this list of Five Essential Questions that you need to ask during an SDE screen. They won't guarantee that your candidate will be great, but they will help eliminate a huge number of candidates who are slipping through our process today.

1) Coding. The candidate has to write some simple code, with correct syntax, in C, C++, or Java.

2) OO design. The candidate has to define basic OO concepts, and come up with classes to model a simple problem.

3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.

4) Data structures. The candidate has to demonstrate basic knowledge of the most common data structures.

5) Bits and bytes. The candidate has to answer simple questions about bits, bytes, and binary numbers.

Please understand: what I'm looking for here is a total vacuum in one of these areas. It's OK if they struggle a little and then figure it out. It's OK if they need some minor hints or prompting. I don't mind if they're rusty or slow. What you're looking for is candidates who are utterly clueless, or horribly confused, about the area in question.

Of course, you'll want to modify this process to reflect the realities at your shop-- so I encourage you to read the entire article. But Steve does provide some examples to get you started:

Coding

Write a function to reverse a string.
Write function to compute Nth fibonacci number.
Print out the grade-school multiplication table up to 12x12.
Write a function that sums up integers from a text file, one int per line.
Write function to print the odd numbers from 1 to 99.
Find the largest int value in an int array.
Format an RGB value (three 1-byte numbers) as a 6-digit hexadecimal string.

Good candidates for the coding problem are verifiably simple, with basic loops or recursion and perhaps a little formatted output or file I/O. All we want to know is whether they really do know how to program or not. Steve's article predates it, but I'd be remiss if I didn't mention Why Can't Programmers.. Program? here. The FizzBuzz problem is quite similar, and it's shocking how often interviewees can't do it. It's a bit hard to comprehend, like a potential truck driver somehow not being able to find the gas pedal or shift gears.

Object-Oriented Programming

Design a deck of cards that can be used for different card game applications.
Model the Animal kingdom as a class system, for use in a Virtual Zoo program.
Create a class design to represent a filesystem.
Design an OO representation to model HTML.

We're not saying anything about the pros and cons of OO design here, nor are we asking for a comprehensive, low-level OO design. These questions are here to determine whether candidates are familiar with the basic principles of OO, and more importantly, whether the candidate can produce a reasonable-sounding OO solution. We're looking for understanding of the basic principles, as described in the Monopoly Interview.

Scripting and Regular Expressions

Last year my team had to remove all the phone numbers from 50,000 Amazon web page templates, since many of the numbers were no longer in service, and we also wanted to route all customer contacts through a single page.

Let's say you're on my team, and we have to identify the pages having probable U.S. phone numbers in them. To simplify the problem slightly, assume we have 50,000 HTML files in a Unix directory tree, under a directory called "/website". We have 2 days to get a list of file paths to the editorial staff. You need to give me a list of the .html files in this directory tree that appear to contain phone numbers in the following two formats: (xxx) xxx-xxxx and xxx-xxx-xxxx.

How would you solve this problem? Keep in mind our team is on a short (2-day) timeline.

This is an interesting one. Steve says 25% to 35% of all software development engineer candidates cannot solve this problem at all-- even with lots of hints and given the entire interview hour. What we're looking for is a general reluctance to reinvent the wheel, and some familiarity with scripting languages and regular expressions. To me, this question indicates whether a developer will spend days doing programming work that he or she could have neatly avoided with, perhaps, a quick web search and some existing code that's already out there.

Data Structures

What are some really common data structures, e.g. in java.util?
When would you use a linked list vs. a vector?
Can you implement a Map with a tree? What about with a list?
How do you print out the nodes of a tree in level-order (i.e. first level, then 2nd level, then 3rd level, etc.)
What's the worst-case insertion performance of a hashtable? Of a binary tree?
What are some options for implementing a priority queue?

A candidate should be able to demonstrate a basic understanding of the most common data structures. More specifically, the big ones like arrays, vectors, linked lists, hashtables, trees, and graphs. They should also know the fundamentals of "big-O" algorithmic complexity: constant, logarithmic, linear, polynomial, exponential, and factorial. If they can't, that's a huge warning flag.

Bits and Bytes

Tell me how to test whether the high-order bit is set in a byte.
Write a function to count all the bits in an int value; e.g. the function with the signature int countBits(int x)
Describe a function that takes an int value, and returns true if the bit pattern of that int value is the same if you reverse it (i.e. it's a palindrome); i.e. boolean isPalindrome(int x)

As Steve says, "Computers don't have ten fingers, they have one. So people need to know this stuff." You shouldn't be treated to an uncomfortable silence after asking a candidate what 2^16 is; it's a special number. They should know it. Similarly, they should know the fundamentals of AND, OR, NOT and XOR-- and how a bitwise AND differs from a logical AND. You might even ask about signed vs. unsigned, and why bit-shifting operations might be important. They should be able to explain why the old programmer's joke, "why do programmers think Oct 31 and Dec 25 are the same day?" is funny.

Performing a thorough, detailed phone screen is a lot of work. But it's worth it. Every candidate eliminated through the phone screen saves at least 8 man-hours of time that would have been wasted by everyone in a hands-on test. Each time an unqualified candidate makes it to the hands-on test, you should be asking yourself-- how could we have eliminated this candidate in the phone screen?

Reinventing the Clipboard

Over time, I've become something of a desktop mimimalist. Sure, I'll change a few settings to my liking, but I no longer spend a lot of time customizing my desktop configuration. I've learned that if the defaults aren't reasonably close to correct out of the box, then the software is probably doomed anyway. For most users the default settings are the only settings.

One of the things I always have to change, much to my chagrin, is the default clipboard behavior. I originally wrote about this in 2005:

In this era of 3 GHz processors, 1 GB memory, and 500 GB hard drives, why is the Windows clipboard only capable of holding a single item? Sure, you have fancy multi-level undo and redo in applications like Microsoft Word and Visual Studio. But not the clipboard. It holds exactly one item. Copy another item to the clipboard and your previous clipboard item is irrevocably lost.

The only improvement since then, sadly, is in the PC specifications. Three years later, we're stuck with the same old single-item clipboard model. The clipboard isn't some obscure operating system feature, either. People use it all the time. There's actually hard data to back this up, at least for Word 2003:

Top 5 Most-Used Commands in Microsoft Word 2003

1. Paste
2. Save
3. Copy
4. Undo
5. Bold

Together, these five commands account for around 32% of the total command use in Word 2003. Paste itself accounts for more than 11% of all commands used, and has more than twice as much usage as the #2 entry on the list, Save.

Granted, we're talking about a word processing program here, but we live in a copypasta culture. I find that even when I'm not writing, per se, I rely on my clipboard throughout the day. The clipboard is so important that Walter Mossberg specifically mentioned it as a negative in his iPhone review:

There's also no way to cut, copy, or paste text.

This is on a phone, mind you. I'm totally with Walt on this one; it applies to all smartphones. I was surprised how quickly I ran into situations where I wanted to copy and paste something on my Windows Mobile phone, but I couldn't figure out how to. It's not a crippling limitation, but it does illustrate how fundamental the clipboard is, even for the smallest of computers.

It always seemed strange to me that applications had to implement their own oddball per-app clipboard queues to spackle over deficiencies in the operating system's braindead "I can only remember one thing at a time" clipboard implementation. We've long since left the days of applications writing their own quirky little file open dialog behind, but it's somehow OK to implement your own wacky clipboard behaviors in Visual Studio, or Office?

If, like me, you'd prefer operating system level improvements in the clipboard, there are quite a few options out there. I've been quite happy with ClipX. After installing this lightweight little app, instead of pressing

Ctrl + V

to paste a single item, you can opt to press

Ctrl + Shift + V

whereupon you're presented with a menu of recent clipboard items, in a nice visual menu browser format:

Your clipboard history is dynamically saved to disk and will survive a reboot, so you can begin to rely on your clipboard as a sort of quick and dirty digital scrapbook. Isn't that how it should have been all along?

I've become terribly reliant on this improved clipboard behavior, so I always install ClipX on any machine I'm working on. It has some additional default clipboard functions that I've also found quite useful:

Ctrl + Shift + G

perform a Google search using the contents of the clipboard.

Ctrl + Shift + N

open a browser and navigate to the address in the clipboard.

It doesn't matter whether you specifically choose ClipX. It's these three key improvements in the operating system clipboard that I think are important:

1. history
2. persistence
3. visual browser

It's a mystery to me why none of the major operating systems have bothered improving the clipboard. It seems entirely possible to add these enhancements without breaking the simple clipboard paradigms that have been around since the days of Xerox PARC.

The Sesame Street Presentation Rule

After being on both the giving and receiving end of plenty of presentations, I now realize there's one golden rule which applies to all of them:

Entertain your audience.

Every slide of your presentation should serve this fundamental vision statement. Is it entertaining? I don't mean each slide has to contain a wacky joke of some kind. Every slide should provoke a reaction from the audience -- be it controversial, unexpected, amusing, or a meditative Zen koan. Prod your audience. Do this not only to keep them awake, but to engage their brains. Deliver a series of short, sharp shocks that jolt your audience into a heightened state of engagement.

Once your audience has engaged with your presentation, that's when you trick them into learning. The very best presentations entertain and educate-- the common portmanteau is edutainment. The archetypal example of edutainment is Sesame Street.

Sesame Street is the second longest running children's show in the world, racking up 4,160 episodes over 38 seasons so far. They must be doing something right.

The show's format called for the humans to be intermixed with the segments of animation, live-action shorts and Muppets. These segments were created to be like commercials-- quick, catchy and memorable-- and made the learning experience much more like fun. The format became a model for what is known today as edutainment-based programming.

CTW aired the program for test groups to determine if the revolutionary new format was likely to succeed. Results showed that test watchers were entranced when the ad-like segments aired, especially those with the jovial puppets, but were remarkably less interested in the street scenes. Psychologists warned CTW against a mixture of fantasy and reality elements, but producers soon decided to mix the elements. A simple dose of cartoon-like characters lets the humans deliver messages without causing viewers to lose interest.

You might think it's patronizing to lift techniques from a television show aimed at preschoolers, but I find that people of all ages need to be entertained to fully engage with what whatever it is you're presenting. That's why your primary goal for any presentation is to entertain. If the audience doesn't walk out of your presentation thinking "gee, that was fun!", then I can practically guarantee they'll remember little about you or your talk. There's nothing more stultifying than walking out of yet another interminable, droning presentation to realize that all you have to show for it is another hour of your life ticked away. If you design to entertain first and teach second, even if your presentation bombs, at least the audience will get some fleeting entertainment out of the time they invested in you.

So the next time you're putting a presentation together, remember the Sesame Street Presentation Rule -- don't forget to add the muppets!

See You at CUSEC 2008

I have the distinct honor of speaking at this year's CUSEC, which runs from today until Saturday.

So what, exactly, is CUSEC?

CUSEC is the Canadian University Software Engineering Conference, an annual conference about the most interesting topics in software engineering organized for and by students from universities across Canada. What makes CUSEC unique is that it is the only software specific conference that targets students. This means that the presentations you'll see at CUSEC will be about things that matter to you, not just things that matter to professional developers. That doesn't mean that the speakers we get are nobodies, either. Past speakers include David Lorge Parnas, Kathy Sierra, Ralph Johnson, Kent Beck, Alistair Cockburn, Dave Thomas, and many more.

They weren't kidding about the impressive speaker list. This year's keynote speakers are:

• Dr. Jeffrey Ullman
• Zed Shaw
• Tim Bray
• Jon Udell
• Dr. Peter Grogono
• yours truly

It's hard to shake the feeling that one of these things is not like the others, but I suppose the conference organizers had their reasons for inviting me, however crazy those reasons may seem to me.To provide a sense of history, the CUSEC keynote speakers from 2007:

• Dave Thomas
• Pete McBreen
• Ralph E. Johnson
• James R. Cordy
• Dr. Venkat Subramanian

And the CUSEC keynote speakers for 2006.

• Peter Grogono
• David Heinemeier Hansson
• Kathy Sierra
• Chad Fowler
• Connie Heitmeyer

I think you get the idea, so there's no need to list the 2005 speakers. It's an honor to be among such distinguished speakers. How distinguished? Many of these folks have their own Wikipedia entries! I had the opportunity to meet Tim Bray today, for example. Since 2004, CUSEC has grown into the premier conference by computer science students, for computer science students-- across the whole of Canada. It's too bad there isn't an equivalent student-run conference for American computer science students.

This is also my first trip to Canada. After many years of wanting to visit our northern neighbors, I've finally made it. I have to agree with William Gibson; as he wrote in his book Spook Country, Canadian cities look the way American cities do on television. I'm enjoying Montreal and the Canadian perspective on life tremendously. It's refreshing, although I am not sure I needed the televised Capital One Grand Slam of Curling.

I wasn't able to find any video archives of previous CUSEC keynotes, but I was told by Edward Ocampo-Gooding, our keynote host, that they're capturing hi-def video of this year's keynotes. Assuming my talk isn't a total disaster, I'll update this post with a link to my keynote when it is available.

Update: 1/11/09 Video of my CUSEC keynote is now available.

Update: 1/23/08 My CUSEC 2008 keynote slides, "Is Writing More Important Than Programming?", are available for local download (ppt, 3mb).

Typography: Where Engineers and Designers Meet

Over the christmas break, my wife and I visited New York City for the first time. One of the many highlights of our trip was the Museum of Modern Art, which is running a year-long special exhibit, 50 Years of Helvetica. It's a tiny exhibit tucked away in a corner of MoMA. Blink and you'll miss it amongst all the other wonderful art. But even a small exhibit provides ample physical evidence that Helvetica-- a humble font, nothing more than a collection of mathematical curves shaped into letterforms-- had a huge impact on the world.

Helvetica is so highly regarded in the design world there's a full length documentary on the topic: Helvetica the Movie.

Another little-known fact about Helvetica is the relationship between this timeless classic and another font you've almost certainly encountered before: Arial. John Gruber explains:

Helvetica is perhaps the most popular typeface in the world, and is widely acclaimed as one of the best. Arial is a tawdry, inferior knock-off of Helvetica, but which, to the detriment of the world, Microsoft chose to license for Windows simply because it was cheaper. Because Arial is a default Windows font and Helvetica is not, it is ubiquitous. Mark Simonson's "The Scourge of Arial" is an excellent resource on both Arial's history and its typographic deficiencies; his accompanying sidebar is an excellent primer on the specific differences between Arial and Helvetica.

You do have to be something of a font geek to appreciate the subtle differences between Helvetica and Arial, much less Helvetica and its precursor, Grotesk. But the discussion leads directly to another hugely important twenty-first century problem: how do you copyright the completely abstract, pure intellectual property that is a font?

All computer geeks tend to fall in love with typography at some point in their careers. Donald Knuth is a fine example; frustrated with the limited typesetting options available for his books in the late 70's, Knuth went on a "brief hiatus" to come up with something better. Seven years later, he unleashed TeX upon the world.

In 1977, Knuth halted research on his books for what he expected to be a one-year hiatus. Instead, it took 10. Accompanied by Jill, Knuth took design classes from Stanford art professor Matthew Kahn. Knuth, trying to train his programmer's brain to think like an artist's, wanted to create a program that would understand why each stroke in a typeface would be pleasing to the eye. "I wanted to try to capture the intelligence of the design, not just the outcome of the design," he says. For example, how do you insert line breaks into a paragraph so there isn't too much space between words and so that most of the lines don't end in hyphens? Although this seems like an aesthetic challenge to be solved by human taste, Knuth says, computers do it well. "This is a combinatorial problem," he explains. "There might be a thousand ways to break a paragraph into lines and each way has a score." His solution was to build a computer program capable of ranking the thousand options and picking the best one.

Typography and fonts are a rare and vital intersection point between software engineers and designers. And there's absolutely no better book on the topic than Thinking with Type: A Critical Guide for Designers, Writers, Editors, & Students. I recommend it without hesitation to all of the above, and certainly to software engineers with even the slightest passing interest in typography.

Like all great books, it teaches you "why", not "how":

This is not a book about fonts. It is a book about how to use them. Typefaces are an essential resource employed by graphic designers, just as glass, stone, steel, and countless other materials are employed by architects. Graphic designers sometimes create their own fonts and custom lettering. More commonly, however, they tap the vast library of existing typefaces, choosing and combining them in response to a particular audience or situation. To do this with wit and wisdom requires knowledge of how-- and why-- letterforms have evolved.

I think I can trace my initial interest in fonts way, way back to the pirate crack credit screens on Apple // software I encountered as a wayward teenager.

I count four fonts on this crack screen. There were countless disk sets of these low-resolution bitmap fonts to choose from. Even back in the mid-80s, these primitive fonts added a particular style, a feeling, an intonation to the text-- and we only had a dismal little 280 x 192 screen to work with. How wonderfully liberating it must feel to have thousands of RGB anti-aliased pixels to render beautiful fonts with today, much less the millions we'll eventually have.