March 16, 2009
Have you ever been to an interview for a programming job where they asked you one of those interview puzzle questions? I have. The one I got was:
How much of your favorite brand of soda is consumed in this state?
And no, the correct answer is not who cares, unless the thing you don't care about is getting the job you're interviewing for. I didn't know it at the time, but this is a Fermi Question. (To prevent spoilers, the answer can be found in a previous blog post.)
Puzzle questions were all the rage in programming interviews in the 90s and early aughts. This is documented in the book How Would You Move Mount Fuji? with a specific emphasis on Microsoft's hiring practices.
It is prudent to study common interview puzzle questions if you know you'll be interviewing at a company that asks these sorts of questions. And if you think you're ace at programming puzzle questions, then I challenge you to point your massive brain at this, the hardest interview puzzle question ever:
A hundred prisoners are each locked in a room with three pirates, one of whom will walk the plank in the morning. Each prisoner has 10 bottles of wine, one of which has been poisoned; and each pirate has 12 coins, one of which is counterfeit and weighs either more or less than a genuine coin. In the room is a single switch, which the prisoner may either leave as it is, or flip. Before being led into the rooms, the prisoners are all made to wear either a red hat or a blue hat; they can see all the other prisoners' hats, but not their own. Meanwhile, a six-digit prime number of monkeys multiply until their digits reverse, then all have to get across a river using a canoe that can hold at most two monkeys at a time. But half the monkeys always lie and the other half always tell the truth. Given that the Nth prisoner knows that one of the monkeys doesn't know that a pirate doesn't know the product of two numbers between 1 and 100 without knowing that the N+1th prisoner has flipped the switch in his room or not after having determined which bottle of wine was poisoned and what color his hat is, what is the solution to this puzzle?
In other words, I hate puzzle questions.*
I'm also not a huge fan of those abstract impossible questions, eg, "how many optometrists are there in Seattle?", but I suppose that's a matter of taste. If you absolutely must, at least ask an impossible question that has some relevance to a problem your very real customers might encounter. I just can't muster any enthusiasm for completely random arbitrary puzzles in the face of so many actual, real world problems.
And yes, I totally failed that interview. Which was disappointing, because it was kind of a cool job.
Not that my proposal for interviewing programmers was any more popular, though I do think it's much better.
I have my own theory about the ideal way to interview developers: have the candidate give a 10 minute watercooler presentation to your team on something they've worked on. I think this is a far better indicator of success than a traditional interview. You'll quickly ascertain:
- Is this person passionate about what they are doing?
- Can they communicate effectively to a small group?
- Do they have a good handle on their area of expertise?
- Would your team enjoy working with this person?
You'd certainly want to complement this type of interview with some actual hands on programming, to make sure the applicant isn't full of crap -- although I'm pretty sure that you can't B.S. your way through a technical presentation to a handful of your peers if you truly have no idea what you're talking about. (And if you can, you should be CEO of a startup by now.)
What I'm optimizing for here is the ability to communicate. Most programmers, once they pass the FizzBuzz level of competency, are decent enough. But coding chops aren't enough. To go from good to great, you must be able to communicate effectively: with your teammates, your manager, the users, and ultimately the world.
My wife and I just finished a five day hospital stay for the birth of our first child. During our stay, we were assisted by a parade of different nurses, at least two different nurses every day, sometimes more as we progressed to different areas of the hospital and through daily shift changes. The quality of care at this particular hospital is generally quite high, but we were flummoxed by the disparity in care between the worst nurses and the best nurses. After a few days, I finally figured out the common characteristic -- the worst nurses were invariably the worst communicators! The fact that these nurses couldn't effectively communicate with us:
- why they needed to do something
- what the options were
- offer advice
- troubleshoot our problems
Meant they ended up feeling like rigid, inflexible proceduralists who didn't care or constantly had to appeal to authority. Of course, this wasn't true. I'm sure they were perfectly competent registered nurses. But in the absence of reasonable communication, it sure seemed
that way. To be fair, these nurses were frequently (but not always!) non-native English speakers.
Hiring is difficult under the best of conditions. But an interview process that relies too heavily on puzzle questions is risky. Sure, you may end up with programmers who can solve (or memorize, I guess) the absolute gnarliest puzzle questions you throw at them. But isn't effectively communicating those solutions to the rest of the team important, too? For many programmers, that's the hardest part of the puzzle.
* although I expect aficionados of the style should be able to identify all the classic interview puzzle questions represented here.
Posted by Jeff Atwood
This like the stupid printf questions I had to de-construct back in the 1980's. The questions have no relavence to the job I do (programming) or to any position I would be interested in. I would likily tell them good luck with with their project and leave.
'I've never interviewed anyone, but I always thought it would be good to ask the person to play Minesweeper on intermediate mode. Tell them that you only want them to make moves that they are certain about - if they can correctly state that there are no certain moves remaining, that would be considered winning.'
I think I could win it before my first move. -- Steve W
You would be wrong :). Windows' minesweeper is guaranteed not to give you a bomb on your very first click. You have a fairly good chance of terminating after that first click, though (especially if you click toward the middle).
I have to say to the military guy: that's kind of a tough difference, because guessing -- that is, estimation given imperfect knowledge -- is pretty much central to any field of engineering*. You get real data when it is reasonable, and you estimate when it isn't. And determining whether getting real-data is reasonable is a sort of meta-estimation in and of itself. That's why I think how many x (are there/can there be) in y type questions actually have real, germane relevance. I guess maybe it's the opposite for most military scenarios, but there it is.
The linked article about the guy walking out of the how would you move mount Fuji interview tells me two things:
1. The interviewer doesn't pass because the interviewee asked back a question that is really similar in nature. Why would you move mount Fuji. If you have to be able to answer How, why shouldn't you be able to answer Why with just as much BS. Maybe an eccentric billionaire has contracted your company because the view from his Fuji-facing mega-tower is blocked. Maybe it's a scheme to realign the earth's magnetic field and spin characteristics.
2. But it all turned out okay in the end because the interviewee was too much of an obstinate asshole and it clearly wasn't going to work.
I don't think completely abstract puzzle questions are really the best way to go, and I've never been asked in an interview to answer a question that wasn't either a real problem (even if well-known and previously solved) or just 1 fairly-obvious removed from a real problem, but I guess it does weed out a few people who have their strong opinions strongly held. Still, I wonder how common these people are in the real world, and how many people who even comment about just leaving are really just bluffing and full of anonymous bravado.
As an exercise to the mind, I like puzzle questions. To some extent they really do tell me something about a person, but I don't know that it correlates very much at all to successful developers.
*Yes, I know, not every programmer is an engineer. Some are, and those are the ones that can justify those questions.
I think I could win it before my first move.
The first square you click on a Windows minesweeper board is certain to be empty. I think the algorithm actually waits for the first square clicked, then distributes the N mines among the remaining squares. This is part your strategy, since you should click a square that gives you the best chance of being able to expand.
Some homebrew freeware versions, like the Palm version, and maybe some linux versions, have the possibility of hitting a mine on the 1st click, but not the original Windows game.
My 1980's programming course interview questions were:
How many piano tuners are there in London?
How many of them are blind?
Is the Pope catholic?
Do bears shit in the woods?
And.. I got the place!
ps I made up two of the above questions.
Thankfully, back when I was applying for jobs potential employers didn't such questions. They were more interested in what you could do, and how cheaply you were willing to do it.
As for Mount Fuji, it's already moving in most frames of reference. To get it to move, one must only wait. But in one special frame of reference, Mount Fuji will never move, no matter what forces one applies.
It would be interesting to here this topic discussed on the SO podcast as Jeff and Joel obviously have polar opposite views.
The more I think about it the more I can see some merit in Joel's point on impossible questions - Smart candidates will realize that you are not quizzing them on their knowledge, and they will enthusiastically leap into trying to figure out some back-of-the-envelope answer. Not-so-smart candidates will get flustered and upset.. But I still think it ought to be possible to achieve the same effect within a more relevant context.
As many of the commentors pointed out, there is no silver bullet. However, there are skills that employers are willing to give employees time to learn and develop during their career.
Communication is one of these, very often, especially for entry level positions.
However, problem solving is (largely) not a learned skill. It is something that is (hopefully) developed within you earlier.
When I interviewed at Microsoft, I answered several puzzle questions but they seemed to be dying out soon thereafter.
One of the more interesting interview techniques I've heard was given by John Robbins: he had people bring in a printout of an example of good code with them to the interview. The subject didn't have to have written the code (could have, didn't matter), but was to be prepared to discuss why they considered it good. It served as a springboard into discussion as well as a glimpse into the interviewee's thought process, without the pressure of having to write good code RIGHT NOW as part of the interview.
Interesting. Your recommendation that interviewees give a short presentation on their past work comes right out of Tom DeMarco Timothy Listers's landmark book Peopleware. In chapter 15 - Hiring a Juggler, they suggest the very same thing. Probably a good idea, but I've never been able to directly apply it in any interview situation.
For interviews I've given, I've never liked the open-ended Fermi questions. They do allow interesting insights into a person's thinking process, but they are too off the wall. There are other ways to approach this.
Although I haven't applied the 10-15 minute presentation idea, you can get some sense of the depth of a person's experience by asking interviewees to talk about some particular aspect of what they've worked on -- what was the hardest code you've written? Tell me about the challenges in the XYZ project? How would you have improved the RST project?
Some interviewees just won't talk. If they can't discourse for 3-5 minutes on some interesting aspect of their work -- they go to the bottom of the pile. If they can't remember key details of some problem they worked on intimately -- bottom of the pile.
One item we did adopt from Peopleware is that we make interviewees code. Part of the interview process is a 30 minute session in which we introduce a moestly difficult problem, and ask them to write code for a solution. The actual code isn't as important as the process of watching them solve it. But, at the very least, we don't hire jugglers without watching them juggle.
I was interviewed for a position at Microsoft and I was asked a puzzle question. Did not answer it, but passed the interview!
Don't forget, if you are going to ask a candidate to write some code, *give them a keyboard* and at least Notepad. NOT a pencil and paper. I hate it when I literally have to *write* out code. That's not how I do it on the job, and I can type a lot faster than I can write. I can't tell you how many times I've had this happen.
For extra realism the test should replicate the conditions the candidate would be working in. Full development environment. If you're hiring me for a MS shop, give me Visual Studio with all the trappings (intellisense, etc).
To continue with the juggler analogy, that's like giving the juggler candidate paper representations of balls to demonstrate with, not the real thing. Yeah, you *could* juggle them, but it wouldn't be the same.
congratulations on surviving hospital, having babies is natural, going to hospital can be fatal!
Following your recent bad apple post, I would have thought a way of assessing how the person makes people feel, let's hope your 'Would your team enjoy working with this person?' question answers this.
Microsoft used to have interviews like that.. not anymore. get a life.1!!
The problem with presenting with past projects at other employers is those damn company secrecy policies
Really? I'm sorry you feel that way. I think presentation is a terrible way to interview programmers. Communication and Lexical skill while both desirable traits are sometimes mutually exclusive. In fact, I get terribly nervous around presentations, but that doesn't mean I don't have passion for my work. I always document my code well, and always have an eye out to be sure my programs are efficient and readable. My management reviews are always terrific. I just dislike presentation. *shrug*
It's not in the employeers interest what you can do, (by the way is all crap)(because if you get interviewed by the executive, he doesn't understand it anyway), but how cheaply you are willing to do it.
The programming skills aren't important at all, it's your social ability and how you communicate..
If you even can't concatenate two strings together that's not so important (beacause I promise you there are a couple of people calling themselves programmers that hardly can do that).
I am non-native English speaker, too....
communicating! communicating! communicating! communicating! ...
such like developers! developers! developers! ...
I always feel anxiety before using English.. Using foreign language is so difficult.
5 days? That sounds like something wasn't quite right at first. Hope everyone is healthy and well now.
That must be a cultural thing.
In France, I had a boatload of job interviews (most of them unsuccessful, of course, my jobless periods happened, unsurprisingly, in crisis periods like now) and as far I as recall, I haven't had such questions.
From time to time, I was asked some technical questions, or to find as much errors as possible in a small C program (did a good job there) or to write a small C code.
But most of the time, the job interviewers relied on resume scrutinizing, asking what I did at various periods, etc.
Now, we have a 3 months (can be extended to 6 months) period where we can be fired immediately (after this period, it takes 3 months to fire somebody), so if you are blatantly incompetent, if you had bluffed about your experience, etc., it will become obvious and out you are. A loss of time for the employer, but not so much, and I suppose most candidates are honest anyway.
how much of your favorite brand of soda is sold in this state?
I hate soda and don't drink it. Next!
@Schmoo, thanks for letting me know about this Tim Minchen person. That was great! A lot more informative than this repetitive prattle about Mount Fuji (To put it in the modern dialect of the American teenager OMG WTF who cares?)
@WordofmouthMike: congratulations on surviving hospital, having babies is natural, going to hospital can be fatal!
The 'natural' mortality rate of childbirth (ie, nothing is done to prevent death of the mother) is estimated to be 15,000 in every million births. In Europe and the US, the current mortality rate is 90 in every million births. A few hundred years ago, less than 50% of women survived the birth of all of their (then standard) 5 or 6 children. But hey, who needs facts when you can just add the word natural to everything? http://www.youtube.com/watch?v=UB_htqDCP-s
One of my favorite questions to ask is Did you notice anything in particular about our facility when you came in? (not applicable for phone interviews of course). Its not a pass/fail question, but it gives a little insight into how they think. Out of the 20 or so interviewees for the last opening here, the ones that didn't notice anything also didn't have any programming skills; the ones that said something vague (that parking was a problem) had basic skills; but the ones that pointed out specific things (like the ashtray 50 feet after the no smoking sign) had some l33t skilz.
This is so fucking full of bullshit i'm very happy i don't work in your company... Can you make a difference between attention to useless details and ability to grasp detail when you actually NEED it. Actually, forget it, this is worse than that. I could basically say the total opposite, a lot of programmers i know don't notice shit about this kind of things unless they actually need to, because they're using their mind on more important matters.
I so hate those tricky shits about how to be the best hirer anyway ...
@Raph, I agree. That is just pretentious drivel. The kinds of comments I see about hiring programmers are always so strange, moronic, stereotypical, and nonsensical.
The first square you click on a Windows minesweeper board is certain to be empty.
Doh! I guess I don't get the job.
The Soda question is easy ...
I live and work in the UK ...
Soda is called either a fizzy drink or cola not Soda
There are no States in the UK
So the answer is : None
Do I get extra points for lateral thinking ....
@WordofmouthMike: congratulations on surviving hospital, having babies is natural, going to hospital can be fatal!
@Schmoo: The 'natural' mortality rate of childbirth (ie, nothing is done to prevent death of the mother) is estimated to be 15,000 in every million births. In Europe and the US, the current mortality rate is 90 in every million births. A few hundred years ago, less than 50% of women survived the birth of all of their (then standard) 5 or 6 children. But hey, who needs facts when you can just add the word natural to everything?
Facts: Women are screened for possible complications for the 8-9 months leading up to the birth, have a fully trained midwife on hand with a complete portable medical kit, and you only an emergency call and a short journey away from a modern hospital (or they would recommend a hospital birth), so a home birth is once again an option....
A few hundred years ago women had no screening, no medically trained midwife, no medical kit, no phone, and no access to hospital... and were often malnourished, a large number died after the birth not during it ....
Jaster said : There are no States in the UK
The UK *IS* a state.
From Wikipedia : The United Kingdom is a unitary state consisting of four countries: England, Northern Ireland, Scotland and Wales.
So ... I guess you don't get the job.
The simplest way to figure out if someone can program is to ask them to write a small program. During the intervew.
Say: Here's a PC (not connected to the internet), I want you to write a program to (pick one):
a. convert a string to an integer without using the library routine that does it for you.
b. backup every file in a given directory and that directories' children to another directory.
In one book I read, back in the days when I was reading books about how to run a development team, the author said something like the way most of us hire programmers is like hiring a juggler without actually asking him to juggle in front of you before you hire him.
Other techniques I've used with success include asking a programmer to bring in a program or part of a program she wrote that she was proud of and discuss briefly what it was that she found most interesting. One candidate brought a 1,000 page print out and couldn't tell me anything about it. She didn't get the job.
My wife and I also just birthed our first kid at Alta Bates (your photo of the footprint blanket is the giveaway) and I must agree that the constant flux of nurses is ridiculous. They even send in head nurses just to ask you questions about how good a job the other nurses are doing! You can imagine I had some fun coming up with the answers that I *didn't* give them.
And I have an unrelated question based on the same post. (Reading your segueways between disparate topics is always fun and has often reminded me of William Poundstone, which is *really* funny because today is also the first time I've seen you mention one of his books..!)
The question is about your interview process. My favorite interview is to ask the candidate about a recent project. (No prep time, but also no audience other than me.) I then make up a way to enhance the product he tells me about, and ask him how he'd build that enhancement. It's a new feature, or an optimization, or a public API for the product, or whatever. This seems like a logical extension of your favorite type of interview, so the question is, have you done that before and was it informative?
By the way, the company I know that asks puzzle questions is full of very smart geeks and they ask puzzle questions simply to gauge whether the personality fits -- not to gauge software skills.
Congratulations on your new son and on making my work partners laugh when I showed them his hello world message.
A little bit OT, but relating to interviews -- well, to job postings that might lead to interviews...
So, what does .NET programming experience really mean?
I feel asking puzzle questions is a good way to judge candidates with no prior experience. Jeff's method will probably work for experienced programmers. That's how I go about it.
I guess I think a little differently than most developers.
The question about moving a mountain made me think about a couple of Zen parables. In one, the wise man answers on bucket at a time. This is probably a variation on the story of a monk who comes upon an empty well at the foot of a mountain and, having only a spoon, starts trekking up and down the mountain to bring melted snow to the well, a spoonful at a time. In either case, the story prmpts consideration of the nobility (or foolishness) in pursuing the futile cause. In the another parable, the ultimate answer to the question is walk away from the mountain. You can't actually move a mountain, but by moving yourself, you can achieve the same practical result.
The are many other references to moving a mountain in various cultures. Faith can move a mountain. If the mountain won't come to Mohammed, Mohammed must go to the mountain. In each, the mountain symbolizes the impossible, and the lesson of the story is found in the value of attempting the impossible.
There are many things that a software developer could learn from contemplating moving the mountain. The least of these is how to actually move a mountain. We do that all the time these days, though seldom to any good effect in the end. Admittedly, I have a bias toward mountain meadows and against strip mines. To my way of thinking, the only valid answer to this question is in the question posed by another, Why would you move mount Fuji.
I have to go now. It is Saturday, and I am spending my weekend getting a website ready for a demo on Tuesday. The site is the product of a two year old project that has meandered through a series of developers without any architectural direction, coding discipline, documented requirements or design, or meaningful software development process of any sort. I joined the project in January. How do you refactor a mess like that? One spoonful at a time...
Interviewers, by and large, seem to be a bunch of useless, unintelligent people, at least the ones who ask these sort of questions. They are trying to look for an easy way out. I have thought about this problem, and come up with the same solution as you have. Ask them about a project they have worked on recently, and tailor your questions to that. Ask what they think went right, what went wrong, what they would do better. Ask about the technologies used, and how they could have used other technologies. Maybe this requires more from the interviewer, they might have to actually think about it, rather than reading something from their printouts. But it might actually stop them getting too bored.
These questions can betray more about the company, especially if it is some small shop.
Puzzle questions suck...
I usually swap puzzle questions with ones that require an answer that can actually be applied to real solutions.
For my team is better to hire a developer with great social skills and good technical experience, than to hire the genius jerk. I have seen projects collapse and sink because of genius-rotten-apple-jerks.
In the end you have to spend more time with coworkers that you spend with your own family. Making sure those guys are likable people is critical for everyone in long therm. Being likable also covers the communication part. Experience has thought us that the main issue in communication matters has to do with trust and not natural communication skills. People how trust each other tend to communicate better regardless of their natural communication skills.
Faith can move a mountain.
Actually Faith would have some difficulty moving a mountain all by herself but her friend Michelle probably could, she's a rather, um, big girl.
If I wanted to move a mountain I'd do it the easy way, just move a molehill and then stand back while my wife made a mountain out of it.
Yep, ms doesn't give these puzzles at interviews anymore. But! Puzzle hunts are regularly held. See the old one here: http://puzzlehunt.isetv.com/.
re manhole covers are cast casting a circular shape is not has hard as casting say a square one.
and 99% of BT's Man hole covers arent round
Yes, I know the really good reason manhole covers are round. It's because that's the only shape which, no matter how the cover is turned or swiveled, prevents the cover from accidentally falling down the hole.
I have sat in on many interviews, and given my fair number, and it is a bitch to find out much about people. I tried a technical test, with a list of questions (like What id the difference between for, do..while and while..do), but this showed mostly that people studies, not that they did.
The last time I interviewed C++ programmers, I asked them to (gasp) write code and (bigger gasp) design a system. I was amazed how hard people crashed. You may not be able to BS your way through a technical presentation, but you definitely can't BS your way through writing real code!
There were only three questions, which required about half a page of answer each. One required real code, one required some sort of design and one required discussion. I am happy with the ONE person out of 5 that actually could give me answers, and he has turned out to be pretty much what we expected from the interview.
(Note. One interviewee, who was applying for a C++ job, asked if he could write the simple program in Pearl.....)
A couple of years ago my wife had to have spinal surgery in Miami and we came to the same conclusion about the nurses. We found that the night nurses were invariably the worst nurses, but they were also the worst people. Rude pushy automatons who obstinately moved through their shift so they could go home. They were atrocious. It kind of makes sense because you'd figure the better nurses would move up the ranks to the day shift. The day nurses were always polite and they at least communicated as if they cared, even if they didn't (I couldn't tell).
I couldn't agree more! We just laid someone off here - a competent programmer, but he failed to communicate what he was doing, and he couldn't understand the communications of others enough to apply them to what he was doing. His work was completely lacking in context, to the point where he wouldn't realize when what he was doing was actually doing harm to the company (say, taking down a production server to work on code directly on the server.) He asked me for a reference - I told him I honestly couldn't give him a good reference even though I knew he was familiar with certain languages inside and out. He just didn't have the communication skills - the minimum level - to make it in the business world.
Probably my number one, top-of-the-list interview annoyance is interviewers who ask something unrelated, or only semi-related, to the position for which I'm applying.
For example, one interview I went to went something like this.
Interviewer: "Are you comfortable with writing MySQL queries?"
Me: "Yes." (I had recently passed the MySQL Certified Associate exam and was studying for the Certified Developer exam at the time.)
I: "Ok. How would you install MySQL in a cluster environment?"
M: Thinking... WHAT?? What does installing the software, esp. in a cluster, have to do with writing queries? Installing the software is barely a footnote in the developer exam, though it's lesson 1 in the admin exam. Does this guy know what he's interviewing for?
Fortunately, having navigated several interviews by now and having marinated in MySQL Developer studies for weeks, I took this in stride and said, "That's an administrator question. I've focused my study on development with MySQL. I'd be happy to answer... "
To refer back to your Phone Screen post, if I was interviewing at a PHP house and they asked me to write something in C++ or Java, I might just walk out (or at least feel like it.)
"what is the solution to this puzzle"?
There is no puzzle, just a statement of facts with no goal stated - as several people good at communicating have already stated.
All others have made various assumptions they weren't asked to make. Those assumptions may be valid, or not, we'd need to ask more questions to determine what the requirements are.
By the way, I teach requirements elicitation and analysis to Master's students at the ANU, so I *better* be good at this kind of thing.