Does Having The Best Programmers Really Matter?

August 3, 2005

Joel has a lengthy entry in which he asks, does having the "best programmers" really matter?

This is something I've talked about before: extreme skill disparity is unique to the profession of software development. The odds of working with a genius or a jackass on any given job are about fifty-fifty.

What's worse is that there's no correlation between experience and skill, either. I've worked with amazingly talented interns and 20 year vets who produced terrible code. Joel provides additional data to support this hypothesis:

The data I rely upon comes from Professor Stanley Eisenstat at Yale. Each year he teaches a programming-intensive course, CS 323, where a large proportion of the work consists of about ten programming assignments. The assignments are very serious for a college class: implement a Unix command-line shell, implement a ZLW file compressor, etc.

There was so much griping among the students about how much work was required for this class that Professor Eisenstat started asking the students to report back on how much time they spent on each assignment. He has collected this data carefully for several years.

I spent some time crunching the data; it's the only data sets I know of where we have dozens of students working on identical assignments using the same technology at the same time. It's pretty darn controlled, as experiments go.

Here's a representative sample for a single CS 323 assignment, showing time spent versus grade/score:

There is absolutely no correlation whatsoever between time spent on the problem and the resulting score for these CS students. It's not much of a stretch to extend this to time spent in the profession of software development.

The rest of Joel's post veers dangerously close to being self-serving-- look at my awesome company and the smart employees I hired! but at least he's finally acknowledging that he is really talking about a very narrow niche:

It's not just a matter of "10 times more productive." It's that the "average productive" developer never hits the high notes that make great software.

Sadly, this doesn't really apply in non-product software development. Internal, in-house software is rarely important enough to justify hiring rock stars. Nobody hires Dolly Parton to sing at weddings. That's why the most satisfying careers, if you're a software developer, are at actual software companies, not doing IT for some bank.

Since Joel's commercial shrink-wrap software development is a tiny percentage of the overall IT market, does this mean that having the best programmers doesn't matter most of the time?

Sadly, I believe the answer is yes.