programming and human factors
by Jeff Atwood
The job market for software developers is hot. This is great news for programmers, but it makes the interview process challenging for potential employers. A reader recently wrote me expressing some concern about the interview process:
You mention Vertigo requiring a code sample, then a phone screening, then a hands-on test in the face-to-face. We have a very similar process, but somehow a large percentage of the candidates who make it to the hands-on test are very poor and should have been eliminated at step 1 or 2. The signal to noise ratio is terrible. It costs a great deal to spend so much time doing face-to-face interviews with people who often should not be developers in the first place. I am curious how much light you might be able to shed on the specifics of your requirements on candidates. What part of the process is the most effective in separating the cream, how and why?
It is very expensive to get the phone screen wrong-- a giant waste of time for everyone involved.
The best phone screen article you'll ever find is Steve Yegge's Five Essential Phone-Screen Questions, another gift to us from Steve's stint at Amazon.
Steve starts by noting two critical mistakes that phone screeners should do their best to avoid:
The point of the phone screen is not for the candidate to drone on about what they've done. The interviewer should push them out of their comfort zone a bit and ask them related questions about things they haven't seen or done before. Ideally, you want to know how this person will react when they face something new, such as your codebase.
In an effort to make life simpler for phone screeners, I've put together this list of Five Essential Questions that you need to ask during an SDE screen. They won't guarantee that your candidate will be great, but they will help eliminate a huge number of candidates who are slipping through our process today.1) Coding. The candidate has to write some simple code, with correct syntax, in C, C++, or Java.
2) OO design. The candidate has to define basic OO concepts, and come up with classes to model a simple problem.
3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.
4) Data structures. The candidate has to demonstrate basic knowledge of the most common data structures.
5) Bits and bytes. The candidate has to answer simple questions about bits, bytes, and binary numbers.
Please understand: what I'm looking for here is a total vacuum in one of these areas. It's OK if they struggle a little and then figure it out. It's OK if they need some minor hints or prompting. I don't mind if they're rusty or slow. What you're looking for is candidates who are utterly clueless, or horribly confused, about the area in question.
Of course, you'll want to modify this process to reflect the realities at your shop-- so I encourage you to read the entire article. But Steve does provide some examples to get you started:
Coding
Write a function to reverse a string.
Write function to compute Nth fibonacci number.
Print out the grade-school multiplication table up to 12x12.
Write a function that sums up integers from a text file, one int per line.
Write function to print the odd numbers from 1 to 99.
Find the largest int value in an int array.
Format an RGB value (three 1-byte numbers) as a 6-digit hexadecimal string.
Good candidates for the coding problem are verifiably simple, with basic loops or recursion and perhaps a little formatted output or file I/O. All we want to know is whether they really do know how to program or not. Steve's article predates it, but I'd be remiss if I didn't mention Why Can't Programmers.. Program? here. The FizzBuzz problem is quite similar, and it's shocking how often interviewees can't do it. It's a bit hard to comprehend, like a potential truck driver somehow not being able to find the gas pedal or shift gears.
Object-Oriented Programming
Design a deck of cards that can be used for different card game applications.
Model the Animal kingdom as a class system, for use in a Virtual Zoo program.
Create a class design to represent a filesystem.
Design an OO representation to model HTML.
We're not saying anything about the pros and cons of OO design here, nor are we asking for a comprehensive, low-level OO design. These questions are here to determine whether candidates are familiar with the basic principles of OO, and more importantly, whether the candidate can produce a reasonable-sounding OO solution. We're looking for understanding of the basic principles, as described in the Monopoly Interview.
Scripting and Regular Expressions
Last year my team had to remove all the phone numbers from 50,000 Amazon web page templates, since many of the numbers were no longer in service, and we also wanted to route all customer contacts through a single page.Let's say you're on my team, and we have to identify the pages having probable U.S. phone numbers in them. To simplify the problem slightly, assume we have 50,000 HTML files in a Unix directory tree, under a directory called "/website". We have 2 days to get a list of file paths to the editorial staff. You need to give me a list of the .html files in this directory tree that appear to contain phone numbers in the following two formats: (xxx) xxx-xxxx and xxx-xxx-xxxx.
How would you solve this problem? Keep in mind our team is on a short (2-day) timeline.
This is an interesting one. Steve says 25% to 35% of all software development engineer candidates cannot solve this problem at all-- even with lots of hints and given the entire interview hour. What we're looking for is a general reluctance to reinvent the wheel, and some familiarity with scripting languages and regular expressions. To me, this question indicates whether a developer will spend days doing programming work that he or she could have neatly avoided with, perhaps, a quick web search and some existing code that's already out there.
Data Structures
What are some really common data structures, e.g. injava.util?
When would you use a linked list vs. a vector?
Can you implement a Map with a tree? What about with a list?
How do you print out the nodes of a tree in level-order (i.e. first level, then 2nd level, then 3rd level, etc.)
What's the worst-case insertion performance of a hashtable? Of a binary tree?
What are some options for implementing a priority queue?
A candidate should be able to demonstrate a basic understanding of the most common data structures. More specifically, the big ones like arrays, vectors, linked lists, hashtables, trees, and graphs. They should also know the fundamentals of "big-O" algorithmic complexity: constant, logarithmic, linear, polynomial, exponential, and factorial. If they can't, that's a huge warning flag.
Bits and Bytes
Tell me how to test whether the high-order bit is set in a byte.
Write a function to count all the bits in an int value; e.g. the function with the signatureint countBits(int x)
Describe a function that takes an int value, and returns true if the bit pattern of that int value is the same if you reverse it (i.e. it's a palindrome); i.e.boolean isPalindrome(int x)
As Steve says, "Computers don't have ten fingers, they have one. So people need to know this stuff." You shouldn't be treated to an uncomfortable silence after asking a candidate what 2^16 is; it's a special number. They should know it. Similarly, they should know the fundamentals of AND, OR, NOT and XOR-- and how a bitwise AND differs from a logical AND. You might even ask about signed vs. unsigned, and why bit-shifting operations might be important. They should be able to explain why the old programmer's joke, "why do programmers think Oct 31 and Dec 25 are the same day?" is funny.
Performing a thorough, detailed phone screen is a lot of work. But it's worth it. Every candidate eliminated through the phone screen saves at least 8 man-hours of time that would have been wasted by everyone in a hands-on test. Each time an unqualified candidate makes it to the hands-on test, you should be asking yourself-- how could we have eliminated this candidate in the phone screen?
I think that the questions with string reversals, malloc, btrees, anything related to hex - n conversion are totally outdated and not productive, because they mesure "math" skills and not the real world knowledge
They are also favoring recent graduates without experiences (because those structural alogorythm tasks are part of exams in my country) and desfavor the experienced developers. I haven't done nothing of the above mentioned things after 1998 when I graduated and I code at least 40 hours per week.
Another thing here to be mentioned is that all of those stupid questions are very well documented on various sites so an smart candidate could memorize all those commonly asked "puzzles" and trick the intervier
To me much better approach is giving a smelly code to candidate to refactor asking them to debug something, asking some of the gotchas we have faced etc
Summarized: IMHO, this style of interviewing candidates is not acceptable and I personally feel somehow insulted when participating in that kind of interviews
@Steve: Um, wasn't that kind of presumed whatever the candidate wrote was inside of a class since he said it was in C#? The class in that instance was of no importance; the METHOD was.
Speaking of interviews, I just came up with a new concept: Have a candidate read and comment on a post just like this one.
Maybe a majority of the commentators here can program their asses off, but personality conflicts can bring a team to a grinding halt as fast as total incompetence. I would much rather take a humble-smart developer on board than a genius-asshole.
Snarky != Intelligent
Ian Hughes on January 24, 2008 3:32 AMWell, I'd be pretty happy with answering all those questions (although maybe a bit weak on Data Structures, and the answers might not always be 'perfect'), but I do think that those questions do cover most of the range of a general programmer. And good developers will have some holes; this is to be expected. There's no such thing as the perfect recruit. Someone who is baffled by all of the questions though, well, I'd avoid.
It's interesting the number of negative comments that seem to be along the lines of "I have no degree / a BA and I don't think these things are important". I did a degree that was more about electronics and mathematics than programming (Cybernetics). When I got hired into my first job, I knew nothing about regexs, databases, uml. So, I very deliberately went and learned these things. And they are useful, even if only used from time to time. I totally get why folks would look for these things in a hire.
That's not to say I'd only hire CS graduates. Computer Science degrees aren't perfect (the students are usually rotten or rarely brilliant at mathematics, for example), and CS isn't really about Software Engineering. But just saying "I don't know these things and I can program, therefore they're not important" sounds a bit like Pointy-Haired-Boss-ism
"I think that the questions with string reversals, malloc, btrees, anything related to hex - n conversion are totally outdated and not productive, because they mesure "math" skills and not the real world knowledge"
Surely tree-related searching is real world knowledge. I mean, it's a core part of search-engines, isn't it?
Yes the debug approach is good as well, but isn't it hard to do down the phone?
I'm sorry but if you cannot show you know at least what the interviewer is talking about and make a stab at the answer, ten what are you doing in a programming job?
The RegEx question which so many people seem to think is irrelevant is basic pattern matching and even if you don't know regex's well (I don't) you should at least recognise that pattern matching is the answer?
The .NET people who seem to think they do not have to understand data structures... How do you choose which objects/routines to use for *efficiency* if you don't know the underlying datastructures? (Although it does explain why some .NET programs are so slow..)
The java library question as many have said should be tailored to the in-house language of choice
But as Jeff said these are not questions to be solved but to weed out the no-hopers who don't understand the question, and don't know how to find the answer...
Jaster on January 24, 2008 4:41 AMAsking these kind of questions can be useful, but it all boils down to what is that you're after. I'd rather have someone who can fix the problems than a mathematician, I have a co-worker who'd be definitely unable to answer most of these, however, everytime we ask him to do stuff, he excels at it or knows when/where to ask if he's blocked. He's got great attitude and is very valuable for the company as such. I guess it's a matter of preference.
I agree with both parties, those who religiously claim that "if you don't know this then you don't deserve..." and then those who bash this saying stuff like "we know how to google".
I understand them both. :) Me? I'd have some trouble with some, but I definitely would be able to fix it and/or find a decent solution to all of them given a box, internet and some time. I wouldn't be afraid to ask or say I don't know this right now.
As Jeff always says: You don't want to the the smartest guy in the room. I'd love to work in a place where I could learn. We are not building a rocket :)
Martin on January 24, 2008 4:44 AM"It seems to me that a lot of the readers of this article have got the wrong idea. It's not like if you don't know this stuff, you will be screened out because you can't program. The idea is to get a feel for the kind of person you are."
Asking these test wil fail to get the good programmers with lots of experiences but let the good student programmers that just came from school get passed the test.
If I get a telephone interview and people ask me these questions. Than I start to have doubts about this company if it is worthwhile to work there. First of all they fail to ask the really important questions how to implement software in such a way that it is easily installable, easily configured, user friendly and when it fails, it does not detonate the computer. The questions about the OO and lists are only minor details in the grand scheme of your software product.
Experience shows that you cannot predict how a program will behave, on paper every- thing works but once you start to implmement and test then some technology that would be very logical and fast to use might nowt work. So the best thing to do is to create a framework that could switch technology or at least make it transient.
They fail to ask how I operate in groups, if I program quick and dirty or create quality code???
I have created very complex stuff that just works reliably and for many years. But I would probably fail the tests and I would probably not be interested in that company anymore if I get these type of questions.
Olaf on January 24, 2008 4:54 AMLet me just say that as a programmer right out of college, I would have had no difficulty proving at least basic competency on those questions. They seem like an entirely reasonable set of quick weed-out problems.
He's not asking you to come up with the regular expression for the phone search off the top of your head - he's asking if you know *what* a regular expression is, and that it can be used for that sort of problem, and that in a day or two you can figure out how to write *one* into your favorite text editor and/or scripting language and figure out how to solve that problem.
After a year of experience (I know, still not much) I've found that the biggest time wastes I've experienced were projects where I or someone else didn't know the right term/concept to begin researching previous solutions. Programmers need to be more than cut and pasters - but they also shouldn't be trying to reinvent the wheel as it will often end up as a Reuleaux triangle.
Notice that he didn't say that they failed the test immediately because they only knew java, or forgot about trees - he said it was an early warning that they will most likely fail other skills questions on the test, based on his experience. I'm entirely willing to believe that a competent programmer could miss a few questions - I know I'd probably miss some of the technical OOP jargon stuff just because they didn't use those names in my classes - but I agree completely failing one area and/or failing multiple areas is a huge red flag.
Mike on January 24, 2008 4:56 AMOne thing Steve mentioned in his original article that we haven't touched on yet is another "failure mode" where the candidate can't remember what they did in their degree two years ago.
Let's see, in my degree I remember ... C programming under Unix, shell programming, functional programming in Lisp, trees, quicksort, multithreading, PC internals, Minix, human computer interaction, and networking.
Haven't used Minix _ever_ since my degree but I can still remember it's a cut down OS and the way it allocates inodes for files does something funny when they get above a certain size.
And that was 15 years ago.
I've hardly ever touched Lisp, but every now again I write or refactor something that makes me think "oh yeah, anonymous functions in C# - they're a bit like lamda expressions."
If you know your stuff, you don't just forget these things!
Ritchie Swann on January 24, 2008 5:06 AMFrom what I can tell, the goal of Steve's interview isn't to find a programmer that is 'the right fit' for the job opening that is currently posted. He's developed something that he believes will find top tier programmers. The kind of people that Joel Splosky is always looking for, or even Martin Fowler (http://martinfowler.com/bliki/PreferDesignSkills.html).
Most professional programmers aren't highly technical, top-tier individuals, just as most basket ball players aren't Michael Jordan. That's just the the facts. You can still use an 'MJ' test to hire an entire team and find out who is your MJ. Whether or not that's a good idea is totally up for debate.
Personally, I prefer the methodology laid out by YerFired above since it gets to the root of the problem and can find the senior staff you need without this kind of rigmarole.
g
Garret on January 24, 2008 5:30 AMThis is a very interesting phone screen technique. I'm certainly interested in using parts of it myself, although I think my application of it would be much lighter.
I guess it depends on the level of the candidate that you're looking for too.
My one concern is that this just filters out those who can't remember their CS 101.
It's has the faint resemblance of a test designed by people who took CS degrees to remove people who don't have CS degress to justify the fact that they spent x years getting their CS degree as opposed to spending that time in the industry.
Both of those choices have their pros and cons and I don't think anyone can categorically state that either option results in better developers than the other. Arguably sure, but i've met idiots with degress and competant people without and visa-versa, so its not so black and white.
There are a bunch of _great_ developers who just _don't_ work in the CS mode.
It seems to me like for some people computers start with bits and bytes but to others they start with a problem of a potential client.
I like solving problems, the quicker and more efficient the better that's why I started coding. Bytes? Sure they're an important underpinning but I rarely need to use them because most of my clients want business software and I use .NET because it wraps low level things up and makes me more productive.
Before the low level community goes up in arms; what is the yard stick for a good developer? I've got software running every day across the world that helps people perform their jobs more effectively, it's clean, well architected and maintainable code. Some of my deliveries have zero bugs. But I've never written a XOR operation in my life.
So do I suck? Is everyone else writing business software in bytes or something?
I sometimes wonder if Yegge's wizards at Amazon actually failed because they wrote the core in LISP and make it so incrediably "good" that no-one could understand it. Is that good or unmaintainable?
Jax on January 24, 2008 5:38 AMThis really rings true for me, and I'd definitely want to screen out the guys who need to Google this stuff. Sure, if you don't know what a hashtable is you can look it up on Google and find an implementation.
But if you never learned what it is, you won't understand its weakness and you'll wind up repeating the beginner mistakes. The worst part of it is the attitude that embraces ignorance. A good programmer is constantly *learning* which means adding to a body of knowledge and applying those lessons learned to make the software fundamentally better, not just perpetually patching over bugs.
ben on January 24, 2008 5:39 AM"Write a function to count all the bits in an int value; e.g. the function with the signature int countBits(int x)"
This was my favorite question of the bunch.
int countBits(int x)
{
return sizeof(int) * 8;
}
Oh wait did you mean all the bits which are set to 1? ;)
Dudley on January 24, 2008 5:43 AMThose questions aren't too hard. I was able to figure out 4 of them with a few minutes. The fibonacci sequence I did as part of brainteaser on the eulerproject. Break string into an array and, recursively pop it back into a string. OO questions I hate because its just more textbook junk. The bitwise is what threw me but I could brush up on it by the second interview.
Usually, when you are looking for work it's good to do a lot of interviews. Each one will make you a stronger candidate. I interviewed twice with the same company. By the end of the first I new I wasn't prepared and told the interviewer that. He kept my CV and called me a few months later. But by then I had found work elsewhere.
The questions shouldn't be treated as an all or none deal. If you don't know try to steer the interviewer towards your strengths. I've let a few interviews get out of hand (example: one diverged in coldfusion because they saw it listed on my CV, but I hadn't used it in years). And try to learn another language.
Joe Chin on January 24, 2008 5:47 AMG'day.
I just took your test as a dare from a mate of mine.
I got 40%. Almost certainly a fail.
So given the fact that I've co-founded a software company (I'm the chief software guy) and we're turning over in excess of 80 million dollars a year, how do you validate the usefulness of this metric?
LittleRascal on January 24, 2008 6:07 AMHere's a better question
You have n trees.
You need to produce a list of arrays of nodes of all combinations of the trees, where only one node can appear from each tree.
How would you approach this problem?
LittleRascal on January 24, 2008 6:16 AMNice post Jeff, spot on. I sent the URL around to my coworkers.
As Remy in "Ratatouille" might have said, anyone CAN program, but that doesn't mean that anyone SHOULD.
John Pirie on January 24, 2008 6:17 AMI think this post misses the point of the interview process. When I interview someone, I concentrate on their interpersonal skills, how they will interact with the team, and what experience they have, what their educational background is, what projects they have worked on, and what type of other programmers they have worked with and how they got along with them. As long as I am confident that they know how to use Google and can research answers then I am confident that they can do the job. As a previous poster said, I'm not looking for a math wiz or savant, I just need someone to develop decent eCommerce solutions and a web application every now and again.
I've been to interviews like this in the past and there wasn't a single question that couldn't be answered with five minutes of work on the internet.
Phil on January 24, 2008 6:18 AMInteresting set of links and articles to peruse. At my most recent job I ended up getting the junior position, when I had interviewed for a more senior one (but I didn't find this out till I came on... nice). Then the senior dev. turns out to be a bully, and a joke. Anyway, I do my thing, and the bottom line is, despite complaining a lot and being picky and demanding, I've just quit that job and gone on my own because the boss is incompetent (as a boss, he's really super nice as a guy) and because the co-worker is a bully.
No big deal, right? Only the catch is, I got a $15/hr RAISE, to WORK FROM HOME, on my OWN CLOCK, and I get to do the COOL PROJECTS. A simple test up-front would have shown who is who (but as this article implies, it takes management being aware of these issues for anything real to be done).
John on January 24, 2008 6:23 AMWell, after reading both this article and the linked article - I'm feeling a little less than super today :)
In any case, I would fair decent on most of the interview, fail on certain points (namely, I code in JavaScript and PHP, not a very data structured environment, likewise, some features of these languages aren't friendly to Java, C and C++) Most of my work is in scripting languages, I'm rusty with ASP classic, can muddle through Cold Fusion (non-tag based code) or grasp it (tag based CF), used to code in Java, took C and C++ in college (even Pascal and had a year of COBOL, but neither ever see the light of day on my resume). A long time ago, I needed to use Perl for stuff.
I don't think I would be totally clueless on everything, but I would provide incorrect answers to questions that, in my scope of programming, would be correct. For example, data structures, I'm familiar with the simple type (array, for example) - not vectors (as all arrays I use scale as I need them) so I wouldn't mention them as I don't use them and they wouldn't come to mind. Likewise, I create objects for complex data structures as I don't have prebuilt ones all the time for what I need.
Bits and Bytes would be fun (as I watch my binary clock and am able to read it...) but some questions I'd know how to do the work, just not necessarily how it is named - although the bitwise operations I've often glanced at but never really figured out a practical use (for what I'm working on) in order to use them. All the logical operands I use all the time.
The 50,000 page phone number lookup would have been easy, I would have mutter grep (not that I've used it, I just know about it) and regular expressions (which I would have provided the code needed for it)
OOP for JavaScript is a slight bit different than for tightly structured languages, I get about 80% of the terms listed on the other article from this blog and can provide examples of them (so, according to the article, I don't know the basics) - but I use OOP coding in my JS all the time, likewise, I'm using OOP in my PHP as well (even if I do use a framework... I do code a fair bit of the needed application on top of that, which uses OOP).
I was surprised no questions on data(base) structure, queries and such. While knowing a high degree of information on low level stuff, knowing how to access and work with databases would, I think, be key in working with large projects and (in my case, my focus is websites) ecommerce websites.
While I do think the list of questions (the concept) is a good one, the specifics of the questions aren't applicable to ever developer job. While some frown upon, look down upon, some of us 'script' coders - we are developers too and our questions should be different from the questions asked of traditional (dare I say "legacy") programming languages. Because, honestly, I see the code turned out from people attempting to use their "hard core" coding on JavaScript and, oh my word, it is the worst code I run across (for example, I have a simple date picking utility for a website form that I've seen written in 40-60k+ of code complex, all classed out and such - my code is a lean 13k, including CSS, which does more than every other similar product on the market - and don't get me started on form processing, can never understand why something so simple is so hard for Java coders to get... but I digress)
I would want to know how a coder would write efficient code (not everyone has broadband, after all, and even if they do, if every "feature" is 60k - the menu, the ajax, the date picker, the search, the CSS, the content slider, the tabs, etc you're 600k - and please, LEARN to use external JS files....)
Kaosweaver on January 24, 2008 6:30 AMFrom Jax on January 24, 2008 05:38 AM
"Some of my deliveries have zero bugs. But I've never written a XOR operation in my life. So do I suck?"
Yes, you suck -- but it's not limited to your lack of "XOR" experience. The fact that you would ever say something to the effect of "Some of my deliveries have zero bugs" indicates to me that you don't actually know much at all about delivering software. (Unless perhaps you count "Hello world!" among your 'deliveries.')
You have failed TWO of the major yard sticks for measuring developers that I use. 1) You believe that you have created non-trivial, yet "bug free" software. 2) You believe in magic.
Allow me to elaborate on #2. There are really only two types of developers: those who believe in magic and those who don't. Your assertion that you don't need to understand the underpinnings of what your software does clearly indicates that you believe in magic. Believing in magic effectively PREVENTS you from solving any problem that occurs on the other side of the magic wall between your code and the code it calls or in the process of crossing the magic wall.
I make it a personal policy never to hire developers who believe in magic, or non-trivial yet bug-free software.
Hi
I'm not sure about your type of questions. E.g. If I Have to evaluate Steve based on the answers he's provided
Look at the implementation of sumFile -- Summing int's into an int? hasn't the man heard of overflows? Not closing an IO in finally?
candidate rejected!
I usually ask questions based on the last project the candidate has worked on. If he can answer them clearly and knows what goes on in his project, he can handle mine. I normally give them plenty of rope at the start and pretend Im not going to ask much. Typically this means the candidate then states how he has taken requirements, architected, designed, developed, lead the team, tested and supported the application. After which i ask pointed questions and If i get an answer oh I didnt work on that area that much after having stated that he did, its a reject. I also always change a couple of parameters on the last project and ask now what changes in your solution?.
Its quite silly going with a fixed book set of questions. E.g. How many of us developing websites really work with bit's on a regular basis?
As Barbie says, math is hard! Fortunately most of the companies I've worked for don't have anyone on staff who would understand any of those questions. The only question I could answer is #3:
3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages.
This is something I've worked out before and I have some code in my notes for future reference. You really need a recursive function to traverse subdirectories and ideally you should generate a report listing of the line numbers and files where phone numbers were found so you can double check your work.
I bet it would not occur to you to load the web in FrontPage and use its search and replace to do the same thing. It even supports regular expressions for find and replace.
Robert S. Robbins on January 24, 2008 7:24 AM""I'm working on a trading system for an investment bank, and I tell you what, the hardest part of being a developer is solving the god damn business problems."
Why is it our problem to solve business problems? Isn't that the job of businessmen? We're trained to be computer scientists dammit! ;)"
No, Damian nailed it. If you work in the business world, you are NOT a computer scientist. You are a business analyst with programming skills. The biggest mistake business developers make is assuming that the BA or the business got it right. They will throw competing requirements at you and not even know it. They will ask you to do things that will cost more than they are willing to pay, and its your job to put it in terms they understand and then build the system they actually want, not the one they asked for to begin with.
razmaspaz on January 24, 2008 7:30 AMGood luck with your interviewing, we went through the same thing here and tried to come up with technical questions to "screen" candidates. It never really worked out for us and our requirements were apparently at a much lower than yours (we were more interested in people who could figure out how to use common productivity tools).
I think the problem is that this approach is exactly the way a programmer would try and go about solving a problem - as if people can be treated like big algorithms. The unfortunate thing I have learned is that this is not really possible, you really need to bite the bullet and spend a lot of time interviewing people, relying on your gut feeling, and then taking a risk on someone. Taking exercises out of a computer science textbook is not really a great assessment of how people will adapt to your workplace.
Matt on January 24, 2008 8:04 AMChaChaChing: After 20 years writing software, you forgot how to write proper English.
Fernando on January 24, 2008 8:08 AMI'm surprised at the level of defensiveness exhibited in the responses here. There are many different flavors of programming job so no one set of screening questions can possibly be appropriate for all jobs. I assumed that questions discussed were relevant to the types of problems being worked on at Jeff and Steve's shops. Am I out in left field for thinking that five years doing ASP.NET does not in itself qualify you to walk right into a job doing Windows desktop apps using C/C++?
I thought the question listed here were dead easy. On the other hand if you asked me about the first three normal forms for databases I'd be sunk. That doesn't mean I'm an idiot, it doesn't mean I'm a bad person. It does however mean that it would be a waste of time interviewing me for a serious database position. I've done database work in the dim past, and I don't doubt I could pick it all up again, but why would you do that when you can get someone who matches all my other stellar attributes, but also has the pertinent knowledge about databases already at their fingertips.
The most frustrating experience I ever had on the hiring side of the desk was at a small consulting firm in 1994. We all knew VB, I knew C as well, and for a variety of reasons we wanted to find an additional person who knew C++. Despite clearly requesting a strong C++ background in our ads, candidate after candidate would show up who thought that knowing C++ meant writing C code but using the C++ operators instead of printf and scanf . Inevitably they would plead that they were fast learners, and they could quickly pick up the rest of C++. We could only sigh and point out that we were clever fellows too and could do the same, but we were really looking for someone who already knew the language, if only so they could teach the rest of us. That experience is why I think these screening questions are useful.
Charles E. Grant on January 24, 2008 8:12 AMHey Now Jeff,
another great post. Makes me think a lot about fundamentals.
Coding Horror Fan,
Catto
Puh-leez this article is a joke. What planet are you on and when are visiting days?
webster on January 24, 2008 8:15 AMJeff, you usually have good articles that hit the spot, but this one was so big "miss" you'd not been hitting barn wall even if you'd been inside of it.
noone on January 24, 2008 8:25 AMI dont think you'd get a code monkey for asking the guy what 2^16 is.
If he takes 3 minutes to think it through fine... but it is an extremely important number. And being able to answer questions like that would get you completely the opposite.
Some of the questions though would get you a code monkey.
But if you put it all together, you have a very well balanced developer that can play code monkey when needed to, and be creative when needed to.
"google is your friend"
Why not apply for a job there then?
hmmmm - not sure that I agree with this one either (I suppose that means this post will be removed as usuall).
I wrote some complicated stuff at University - but generally in my day to day business life, I try my hardest to make things as simple as possible.
KISS
orange, orange, orange, orange!
Steve on January 24, 2008 8:50 AMI just love when an interviewer uses a set of terms that are arcane, part of a general pool of terms developed internally at the particular company, or platform-specific, and expects an interviewee to understand what they're talking about without any further explanation. In one particular case I gave up answering many of their questions simply because I was sick of listening to questions using Windows-only terms that were almost as old as the OS itself, and haven't been used in most of the documentation since before I started writing software. In the end, I found that I really didn't want that particular job, and was glad that it wasn't offered to me.
The best part, though, was that I was able to recognize many of the questions as being minor adaptations of the crap floating all over the internet. The reality of the situation is that I probably would've been able to answer every single question just by reading blog posts and asking for different terms (if they could have provided them).
What I was really looking for was a job that was more focused on software development and that would understand that their particular work has a specific focus that may not be the focus I have had in my work. I would expect questions that allow for some room to figure things out and even the chance to look them up. What I usually find, though, is an escort to a room with 2 senior programmers that have been working in the same field far too long and expect everyone they work with to use the same language they do, and do most of their programming on paper or whiteboards.
Vizeroth on January 24, 2008 9:25 AMI see we have a mix of opinions about these questions. Let's list some pros and cons:
PROS:
- They test the candidate on the RIGHT variety of technologies / skillsets (although I am worried about the lack of database questions), as opposed to HR's idea of "do you know language X?" with the accompanying oddball, irrelevent, googlable questions. (I was asked in a phone screen, by HR, what a transient variable is).
- They indicate the value of an education (As a person who graduated Cum Laude, I don't want to work with someone who doesn't value my education)
CONS:
- They assume that all programmers must be a jack-of-all-trades.
- It omits the most important question: "Why do you want to work for us?" This question should weed out 99% of all candidates. Some might argue that this is a question is an in-person question, but asking it in the phone interview gets the candidate thinking about the commitment involved. It also indicates how much research was done into the company before applying.
I haven't participated in interviews, but I talk frequently with someone who does. What people are really looking for is how much you want the job. I have answered all of these questions in interviews and still not gotten the offer, because I wasn't acting like I wanted the job.
AA on January 24, 2008 9:28 AMFrom Ian on January 24, 2008 07:16 AM
"You have failed TWO of the major yard sticks for measuring developers that I use. 1) You believe that you have created non-trivial, yet "bug free" software. 2) You believe in magic."
Oh great so now i'm a liar too huh?
To account both of these deliveries were applications for mobile devices, one for stock taking the other was a major upgrade for a warehouse solution. They weren't Enterprisy but they weren't trivial.
Basically a big job for one man.
I was as surpised as you sound, I also believe there is no such thing as bug free code and the longer the phone didn't ring the more I worried, but it can happen and it DID.
I don't presume my code is error free which is why I have unit tests and also vigourously test before committing changes to my source control.
Unfortunately you just seem incorrectly assume I'm lying much like the people who incorrectly assume programmers suck if they don't know how to XOR.
Another risk in this scenario is letting your own ego get in the way of hiring a great candidate.
I'm amazed at some of the replies on here.
'I don't need to know about data structures, because I write .NET'
Huh?? Take a look in System.Collections - see those classes, they are data structures.
If you don't understand them then how can you choose the most appropriate one to use?
'I don't need to know binary because I code websites'
Uh-huh.. so if you need to query a database that uses a field containing flags then you are going to be lost?
Or if you need a function that masks together its parameters (like Open(Create|Read|Write)
'I don't know regex so I'd write a throw-away program'
So you have heard of regex, but rather than take an hour to go away and read about it you'd spend a couple of days writing your own broken version.
And 'I use Windows not Unix' is not an excuse - the Find option in most decent text-editors can use regex.
I do agree though that some process-oriented questions would also be good: describe unit testing; give a typical project life-cycle; etc
GrahamStw on January 24, 2008 9:37 AMSigh. For a few minutes at first, I was reading the first piece of advice as "Don't let the candidate drive to the interview." Er ok, I thought, if you think that's important...
SM on January 24, 2008 9:49 AMI like the idea of asking indirect questions about tools and practices. IHMO a good programmer will have reasoned opinions about the tools he uses, the style in which he codes, and the process by which he codes. That can't be the only thing you quiz, but you can learn a lot by asking "Tabs or spaces?" and listening to the justification.
Aaron on January 24, 2008 9:56 AMThese types of posts always seem to bring out the "best" in people.
Anytime I read a post regarding the types of questions to ask in an interview, the subsequent replies devolve into a pissing match. Almost nearly as fast as a ".NET vs Java" or "VB vs C#" flame-war of zealots.
This speaks to both the hubris of many developers and the complexities of what it means to write software. There are no "magic bullets" for interviewing because the needs of various positions differ so damn greatly.
Imagine an advertisement that was titled "Pilot Wanted". Ok, fixed wing or rotary? What type of aircraft? What a "pilot" is can mean a great many things, but any pilot should understand "altitude", right?
Steve Yegge's and Jeff's point is that there can unifying questions; so pick and choose what is applicable to you in a thoughtful manner and f*ckin' save yourself a lot of wasted interview time.
I'm glad you corrected yourself, Jon. I was about to put that code into production :P
Ian Hughes on January 24, 2008 10:04 AM
been on both sides of the phone. Basically, you have a staff position to fill, a contractor is too expensive, management wants to hire Steve Wozniak at the guy on the corner holding the "Going Out of Business" sign's salary.
You have approx. 200 applicants for the position (once you've weeded through the kids of people who attend church with your managers - if you wanted a Facebook page or someone to type your questions into Google all day, hoo-yah, but you're just looking for somone to mentor), all of whom are (according to their CV) dynamic proven self-starters with long tails of success at jobs working at the print shop and university computer lab.
Pop quiz - what do you do? what do you do?
Keywords! Everyone comes up with some questions they think are interesting - after 8-10 years of practical software development experience.
Then you surf the web and come up with a few more, collectively cooing "ooh, that's a good one" because the idealistic side of you wants someone bright-eyed and dedicated to lapping at the oozing pores of your wisdom, not considering that pretty much everyone coming out of college has just spent 4+ years alternating between learning how to have adult relationships, trying to cope with the expectations of their parents and working meaningless jobs while getting the entire collected history of humanity jacked into their heads Matrix-style.
So, you need to weed. Most of it is personality - you want someone who's not an ass, someone who can look at a set of poorly understood facts in miniscule timelines and solve the problem at hand without creating too many additional ones. It would also be nice for them to admit when they are wrong and show you when you are wrong without turning every discussion into a deathmatch.
Plus, it would be cool if they liked Golden Tee and anime and debating who would win a battle between the Death Star and Battlestar Galactica.
So anyway, now you have three goals in your little "interview this person" user story:
1. Conceal the fact that your system is a 2001 value proposition with a tree-ring like series of patches, fixes, stuff a contractor dude did during a 30 day contract, good ideas unrealized, bad ideas overmaintained, things people forgot to maintain and semi-tested code layered on top;
2. This person will not in fact be doing the job that's advertised, but maintenance and support work until s/he proves themself capable of wearing the big-boy pants and not ruining another one of your weekends with an untested patch;
3. Gently ease them into the idea that this is a job, which means you lose the right to expect justice when you cash your first paycheck. There's a great group of people, who are smart and dedicated and often leave you dumbfounded with respect for their intelligence and professionalism. You'd really like to bring someone new in and teach them some good things they could build a career on, but always remember that this isn't really a family and if the stock price tanks or some VP needs to have an accomplishment on his PMI you can and will be out.
So, they come in. It's cool when they dress up (at least to me) - not like we all hang out wearing ascots and drinking martinis but it shows someone is trying to tell you they take the job seriously and they want you to take them seriously. If I want to spend time chatting with a guy in Dockers and a muscle t-shirt I can play GRAW 2 with my sister's idiot boyfriend.
Of course, you're in the middle of a fix because Mr. Dipsy J. Doodle once again checked experimental code into production (and he's in a different reporting org so you can't properly beyatch-slap him, just write a five-paragraph email both managers will delete at the first detection of obtuse techie-speak acronyms like CVS and FUBAR) and you didn't have time to prep the great story about how this is a fan-fricking-tastic place to work.
You'd really like to ask "Hey, I have this process that keeps terminating on Wednesdays when the outside temperature goes over 60 degrees - it's a J-21 implementation of the RBVA-16 use case that tries to implement a singleton strategy pattern factory because that was what the architect we brought in to catapult a design specification at us had read about on the plane ride here" but you'd spend seven hours explaining the business problem.
So you have keywords. Because it's objective and easy and you have to get the pool down somehow so you can get back to work and fix the build in time to have dinner with your wife before the 10 o'clock news.
The respondents fall into three classes:
1. People who try to rephrase the first Google answer that came up;
2. People who sit blankly and say "um..I don't know."
3. People who say "I haven't worked with that, but I would try Google."
So...who would you hire? Is this where I reveal my secret sauce, my magical ingredient that lets me pick out the cream of the crop so we can all be entrepreneural and start our own blogs about developing turnkey Joomla solutions for non-profits saving babies in Afghanistan?
No. It's not. Because again I have no idea if this guy is a twit or a blowhard or what. I know s/he's inexperienced. So I ask, how would you find out? Have you ever dealt with problems like this before?
And then they fall into three subgroups:
1. People who are curious about everything;
2. People who are curious about what they think they are paid to do.
3. People who are basically not curious, either because they are obtuse or they think they have all of the answers worth knowing.
I prefer 1 but I have been through enough to know that I need 1 and 2. I do not want 3. If I have too many 1s, we have a great time and loads of fun and no work gets done. If I have too many 2s, there are a ton of related problems in Bugzilla you have to threaten people with termination/death to find root cause and take CA on. If you have too many 3s...well...that doesn't last.
So the tech questions are a gateway to the thinking process. And finding out if you have any chemistry with the person, which in the end is what hiring is all about.
and all of you outside-the-boxers who want to be snitty about corporate software - I'm sure your new site kicks ass, whether for the Taco Hut or that new social networking board that you and your 20 friends use came up in 5 days with free tools, but I notice you don't trust your investments, airline reservations and bank balances to the hipster doofuses of the world. And trust me, the affiliators and link managers of the world laugh like hyenas at most of your sites coming up in the affiliate request lists.
When it comes to money, or anything meaningful, you want Gollum in the cave and you want someone to be beating him with a whip while he minds your gold. That's the consistent side of the business.
collator on January 24, 2008 10:10 AMOne of the problems with phone interviews is that they don't replicate the process of sitting down and solving a problem very well. I look at programming as like constructing a building - you need tools. You bring many things with you - prior knowledge, experience, knowledge of coworkers (if you asked them a question for example), and various other "tools" such as books, reference materials, and yes, google. It's not the same as being asked a question and expected to answer right off the bat with nothing but what's in your head. It proves knowledge, not ability or talent. Any indication of ability or talent you get from the answers is just a projection of how that person sounds, not an actual test of their aptitude in those areas.
As a psych major I studied interview processes and an interview is the worst predictor of how well someone will do on a job -- think about it, you're testing someone's ability to interview well, or in this particular case, testing their ability to recall specific bits of knowledge. I think that's why some people have such a negative reaction to it -- because if you're asked to prove your knowledge, it makes you look stupid if you don't happen to have the answer.
The "If you don't know this you aren't a good programmer anyway" responses come off as elitist and condescending to me. Sure, studying for an interview proves that you're willing to put effort into something, but maybe some people don't have time to sit around and code and work on brain teasers in their spare time -- especially busy people, who have 40 hour a week jobs and families and lives. In other words... more well-rounded people.
I totally agree that for some jobs you need geeks, especially places like Google and Amazon (although I'm sure they also need people with good team skills and problem solving abilities)... some companies just don't necessarily need that; other skills are more important and should be evaluated for instead. Kudos to the person who said as much earlier, and that companies will in the end be taking a risk on someone and going with their gut. That's often more important for MOST companies, than looking for the egghead. But as always it depends on the task. Sometimes problem solving is more important, sometimes it's creative thinking or 'outside the box' approaches.
"Write a function to reverse a string.
Write function to compute Nth fibonacci number."
Just for kicks I wrote my own versions of these in C#, and then again in Scheme. It was actually a really fun exercise to do the latter! Maybe an interview question worth trying would be to have the candidate quickly learn Scheme (it takes all of 3 minutes, a href="http://en.wikipedia.org/wiki/Scheme_%28programming_language%29"http://en.wikipedia.org/wiki/Scheme_%28programming_language%29/a) and then write a few of those in functional/recursive style.
Anyway I never thought of the interview from the angle presented here, that of "find something they DON'T know and see how they fare," but it makes a lot of sense. I actually interviewed for a job about a year ago and I did pretty well. After being asked all sorts of cool questions about some non-typical binary tree stuff, I was asked to reverse a linked list in C. For some reason though, I completely bombed that question! The guy interviewing me obviously knew that this was something that could be found quickly through an online search and wouldn't actually stump anyone on the job. But, I screwed it up (I was *close*), and I admitted as much, so maybe that was fair enough. I just thought it was funny that I tripped over such an elementary question! (Yes, I did get the job)
Rick Brewster on January 24, 2008 10:33 AM@collator, I think you are exactly right. A lot of the "screening" questions seemed designed to select those who know the same trivia as the interviewer.
You want to hire people that have a spark of intelligence in their eyes, a curious mind. And if they fit in personality wise, to make the team greater than the sum of it's parts, bonus.
juice on January 24, 2008 10:49 AMWhere are the jobs with interviews like this? I have had 3 programming jobs, out of my 7 total jobs in 9 years of IT work experience, and I have never encountered a phone screening, or even a skills test. Like many here, the samples provided in the linked posts seem trivial, and I have trouble (or would have, prior to reading thedailywtf) believing that someone out there who cannot pass these tests is holding a job as a programmer. I would love to go through such an interview process, if only for the novelty and fun of it.
sparr on January 24, 2008 10:51 AMthose things don't define a good programmer - it's the research skills and coding experience that really matter, not if I can recite some BS to you - I agree with David
" It's not the same as being asked a question and expected to answer right off the bat with nothing but what's in your head. It proves knowledge, not ability or talent. "
I want ability and talent from my developers, not the memorization of how to compute Nth fibonacci number - your theory and methods are way off base - good luck in the real world
joe on January 24, 2008 11:07 AMOMG, Steve would be also rejected by me if I saw his 'main()...' C/C++ program.
The standard says 'main' has to return an 'int'.
The phonenumber thing cracks me up, it's a very realistic "problem". I've had to solve it multiple times (for different kinds of data) I don't think it ever took me more than 15 minutes to write a usable regex for anything that can be expressed in one of a few ways and will be on one line.
I guess what I'm saying is that in a worst case scenario it'd take about 20 to 25 minutes for a real programmer to have a working tool to extract the requested information ready enough for testing by someone other than the developer that wrote it. Anything over 45 mins would be unaccaptable i think, even without regexes it's easy enough to do afterall.
It took me under two minutes to write /(\([0-9]{3}\)|[0-9]{3})(\s|\-)[0-9]{3}\-[0-9]{4}/ off the top of my head as a probable solution
(xxx) xxx-xxxx and xxx-xxx-xxxx right here in this textbox, so untested and probably with at least one typo, but given an IDE that should mean the regex is probably very doable in under 5 minutes.
The problem is simple enough to make me think its possible to do it with just sh, more, grep and some piping between them on a single command line.
Anyway, just my 2cts. good post btw.
@Lawrence,
Your post is a great reminder to read comments before writing my own. You are so right.
kris on January 24, 2008 12:18 PMApart from the technical questions I would also ask something about the company and the company values. I had an interview once were they asked me to arrange some company value words. It was a great way for me to know if I would like to work there or not.
Fresh out of school, I failed a bit on the technical questions (that was something like you mentioned) but got the value words pretty nice lined up. I guess I'll apply again soon ^_^
Ola Lindberg on January 24, 2008 12:20 PMIt seems to me that a lot of the readers of this article have got the wrong idea. It's not like if you don't know this stuff, you will be screened out because you can't program. The idea is to get a feel for the kind of person you are.
Obviously if they want someone to come in and get a specific job done then not having the skills in that particular area is a bad thing. But if they are looking for a long term employee then they are going to want someone who can learn and adapt to new stuff quickly. Having the skills and having the ingenuity to learn the skills are two different things.
I've not really bee working that long and haven't come across 'Scripting and Regular Expressions', so i have no clue about that phone thing. I could guess, and I’ve have an idea of where to start looking and that’s what I’d tell my interviewer anyway.
Besides i could always look up some comments made about a article blog about programming. :D
JimmyDeemo on January 24, 2008 12:49 PMIt is interesting that no one has talked about the setting of a phone interview. I rank them just above telemarketing calls. I've done lots of interviews face to face as the interviewer, but never had to do a phone interview as one (the company had buzzword jockeys for that). Yet I did a lot of them as an interviewee during a year long job search. Many of them came while I was preparing dinner for my family. One in particular came on my cell as I was coaching my son's soccer game. The interviewer refused to call back, and since I couldn't give any focus to the questions, I refused to waste his and my time and said good bye. A face to face is scheduled, and I make it a point to leave my cell phone and any other distractions behind. Phone calls are ad-hoc, and don't account for distractions. I only passed one phone interview, and that was for a job I didn't think I was qualified for. I thought I did well on several given the fact I was fielding questions from the interviewer and my kids, plus getting other calls.
Tim on January 24, 2008 12:58 PMIs that the spark fun cell phone ? Very cool.
jd on January 25, 2008 1:00 AM@collator - nicely put...
I enjoyed that, thanks :)
Albeit cynical, its true.
Jonny on January 25, 2008 2:20 AMLet me tell you the "Dec 31 vs. Oct 25" joke is just not 'funny'. Ask your mother, she will explain why. That kind of joke falls within a 'geek' culture that separates programmers from the rest of society... I don't find that healthy.
Been a coder for 20 years, working with programmers for a good while, I encourage my (often younger) colleagues to "not-be-a-freak"... For the sake of our mental sanity!
Now shut down your freakin' computer, get out have some beers, and talk about non-computer stuff, ideally with people not interested in computers at all. In other words, get out of the box! That improves your thinking skills, the one-and-only to be asked for at any job interview. Alas, I am yelling that to myself!
gatopeich on January 25, 2008 2:21 AMI doubt more than 10% of our development staff, myself included, could actually deliver the goods on such a phone screen. If we were lucky you would be able to tell from how we handled ourselves that we were good candidates, but the whole premise here was to get past those gut feelings to an empirical test.
I give it a C-.
Noah Yetter on January 25, 2008 3:20 AMI would happily replace all of these tests with a test on coupling and cohesion. If someone doesn't understand the importance of these two concepts then they have had a deprived education. That is what destroys 99% of projects.
Also, what about a list for hiring managers? They can really suck too.
Jonathan Parker on January 25, 2008 3:58 AMJeff, I will bend the context of your article a bit to pretend that the definition for Q3 was simply "3) Scripting and regexes. The candidate has to describe how to find the phone numbers in 50,000 HTML pages" with no further definition supplied (a better test, I think), as I want to make a point about the assumptions that lie behind the questions (and the possible answers)...
Candidate 1: That's trivial! I can do that in 2 minutes! (Failed)
Candidate 2: I've never had the need to use regexes before, but, hey, that's what books are for. But first, I have the following questions.... (Passed)
Why? Because Candidate 1 will give the appearance of being brilliant and producing quick solutions, but his solutions will probably be rushed, half-assed and ill-conceived.
Candidate 2 asked the following questions during the interview, thereby demonstrating that she understands real world problems and client needs. She is a capable solutions developer, not merely a programmer with a collection of theoretical tools and no practical understanding as to how they might be used...
* How many of the telephone numbers are badly-formed, broken over multiple lines, or protected from harvesting by Java script or embedded as graphics files?
* Are any of the pages Unicode and/or in other languages?
* Once we determine the proportion of telephone numbers that cannot be extracted by regexes, what is an acceptable loss? Are we willing to lose all or them, none, or somewhere in between?
Now, drawing liberally from one of the comments above, the difference between the guy who took days/weeks and the guy who took 2 minutes is that the guy who took longer recovered 99% of the telephone numbers and the other recovered 25% (only properly formed numbers located in the USA stored in plain text).
If this were a privacy issue, for example, "we are required by law to not allow telephone numbers to be published or people can sue us into oblivion (or we can end up in jail)", the yield is important. And, as we are bending the context for the sake of argument, we also assume Amazon, being an international company, wants international solutions to international problems and does not want programmers who are thoughtlessly US-centric (a plague infecting many international websites).
Now, if neither guy asked any questions about the scope of the problem (and it's goals) before starting, both are idiots. One may have produced an inadequate solution that comes back to bite the company 3 years later with a massive lawsuit, the other may have wasted time and money unnecessarily by over scoping the problem.
Before anyone says, the scope of the problem is defined by the assumption that regexes are to be used: it is very common for employers to be determined to use Product/Method A to solve the problem (they only have a hammer and all the world is a nail) and it is up to you, as a seasoned professional, to advise them that Product/Method A may not be up to the job and Product/Method B should be used instead and explain why, clearly weighing up the pros and cons.
Sadly, you could eliminate a lot of candidates by asking: Do you pass parameters, or use global variables?; Draw me a data model for a database that holds questionnaires and their results; Do you know what a For-Do loop is and why you would use it?; What is a class?; What does 'event driven' mean and how does it affect the way you design your code?
I have met a disturbing number of in-house developers that would fail these questions.
"Watch out for one-trick ponies. Candidates who only know one particular language or programming environment, and protest complete ignorance of everything else, are a giant red warning flag."
The counter to this is obvious: those who know many are experts in none. Sometimes you want experts, not dabblers.
Paul Coddington on January 25, 2008 4:11 AM@YerFired:
You said,
Q3: You discover that performing the task takes 100% CPU as well, now what?
The answers I would look for are:
c - here I'm thinking about farming out the work to more than 1 computer, and different computers need to cooperate. It can be as simple as a database, maybe it's more complex
That's an interesting thing to look for. I would have looked for something like this:
"First, I'd want to know why it was taking 100% of the CPU's resources. Then I'd address that. If it's addressable by hardware or software, we could fix that. Either way, we profile the application to determine which parts are consuming the most processor cycles and optimize the heck out of them. If that still doesn't work, then we look at the webfarm solution."
Can the developer see the BIG picture? Is there a problem with the server hardware? Does the problem occur in both the test and production environments? If it doesn't, there's something different between them, and it needs to be resolved. If it's not a server configuration issue, then you profile the application. No line of business application should consume 100% of CPU cycles. HUGE RED FLAG.
Mike Hofer on January 25, 2008 4:49 AMAaron G sounds like StrCpy() guy to me. This was the guy that decided that the best way to determine if ANYONE knows how to program is to check their knowledge of Strcpy:
http://davidavraamides.net/blog/2005/05/07/strcpy/
After all, ANYONE who knew how to program would have read this specific book and studied this specific function and be able to regurgitate it 5 different ways!!! Right? This guy is so convinced that everyone is thinking exactly like him, he isn't able to step outside his own shoes and look at it from the standpoint of a younger person. Similar to you I think, especially with the mocking responses you had to anyone who legitimately questions the validity of certain questions.
Look, you've probably been frustrated by clueless coworkers like most of us, but what I think you fail to see is that THESE INTERVIEWS DO NOT WEED OUT ALL OF THOSE PEOPLE - simply the less clever ones. Believe it or not, you're going to be working with people whose ability and knowledge frustrates you on some level. Trying to design a gauntlet that will make every person hired perfect is not possible - and yes, everything is subjective, but you want to have a reasonable test. You cannot tell me that anyone who knows how to program knows recursion - what about 20 year veteran COBOL programmers who have designed massive systems that didn't use recursion? The real question is, what does the ability to program entail? It's a worthy question and should be considered when designing good phone screens.
"A lot of people seem to think that perfect answers are needed to all these questions; note that this is a first-stage phone screen, just to decide whether or not to bring you in for interview."
You are right about the fact that it is not about the correct answers but how people respond. But if you notice in this thread, the response will be that you will lose good programmers if you even try it on them. In theory it should work, but in praktice, if someone asks me these stupid questions, then the first thing that comes to mind is that the people in that company are stupid moron's so I won't even bother to procede with the interview because working there will limit my skills in a some kind in mind-killing boring serial work.
Good developers do not think inside the box. They dare to go outside the box too, and try to experiment. Otherwise you just create another clone of a program that has been created a zillion times.
You need innovation. I always use Google since I am creating never before used softwaretechnolgy and maybe someone out there has some good idea that I just take and make it even better.
Olaf on January 25, 2008 5:54 AMJust get by the "FizzBuzz" question in the phone screen and I'm willing to talk in person. My most recent problem is the human factor of hiring freaks with "insane" personalities ticks that don't emerge until three months after you hire them. Yikes! How do you interview for that?
Mark Lindell on January 25, 2008 5:56 AMUmm if the scripting question is soo easy to answer (find phone numbers in 50,000 pages of HTML) could someone explain to me how you would do this. Not that I am going to an interview anytime soon but I feel slow considering many thought it was easy.
billybob on January 25, 2008 6:11 AM"We all knew VB, I knew C as well, and for a variety of reasons we wanted to find an additional person who knew C++. Despite clearly requesting a strong C++ background in our ads, candidate after candidate would show up who thought that knowing C++ meant writing C code but using the C++ operators instead of printf and scanf ."
I had the opposite experience, they asked a C++ programmer, but they thought that C++ meant USING the classes from MFC while programming in C. They actually forced me to do it their way since they did not understand what I was doing. Inheritance? Polymorfisme? Private?
Olaf on January 25, 2008 6:26 AMI dont think you'd get a code monkey for asking the guy what 2^16 is.
18.
I have seen a NASA UAV collision avoidance algorithm coded in C which used ^ for exponentiation, though that was in a research paper rather than production code.
Pete Kirkham on January 25, 2008 6:36 AMAnybody else taken a crack at LittleRascal's problem? Good problem, harder than it looks, here's my (probably inelegant) solution in Scheme. I went for stack-intensive rather than having to do any copying/flattening of the data structures, etc.
I'm assuming your trees are binary, for simplicity, and defining a tree as a list with (data left right).
(define global-combos '())
(define left
(lambda (tree)
(car (cdr tree))))
(define right
(lambda (tree)
(car (cdr (cdr tree)))))
(define data
(lambda (tree)
(car tree)))
(define solve
(lambda (tree rest-of-trees current-combo)
(if (null? rest-of-trees)
(begin
(set! global-combos (cons (cons (data tree) current-combo) global-combos))
(if (not (null? (left tree)))
(solve (left tree) rest-of-trees current-combo))
(if (not (null? (right tree)))
(solve (right tree) rest-of-trees current-combo)))
(begin
(solve (car rest-of-trees) (cdr rest-of-trees) (cons (data tree) current-combo))
(if (not (null? (left tree)))
(solve (left tree) rest-of-trees current-combo))
(if (not (null? (right tree)))
(solve (right tree) rest-of-trees current-combo))))))
Then you do such as:
(define trees
'(
(4 (2 (1 () ()) (3 () ())) (5 () ()))
(b (a () ()) (c () ()))
(Z (X () (Y () ())) ())
))
(solve (car trees) (cdr trees) '())
global-combos
((y c 5) (x c 5) (z c 5) (y a 5) (x a 5) (z a 5) (y b 5)
(x b 5) (z b 5) (y c 3) (x c 3) (z c 3) (y a 3) (x a 3)
(z a 3) (y b 3) (x b 3) (z b 3) (y c 1) (x c 1) (z c 1)
(y a 1) (x a 1) (z a 1) (y b 1) (x b 1) (z b 1) (y c 2)
(x c 2) (z c 2) (y a 2) (x a 2) (z a 2) (y b 2) (x b 2)
(z b 2) (y c 4) (x c 4) (z c 4) (y a 4) (x a 4) (z a 4)
(y b 4) (x b 4) (z b 4))
This took me a few hours of muddling, and were I posed this question in an interview I likely would have managed to eke out a few things about "recursion" and "closures" and some graphs on a piece of paper, and probably the start of some incorrect but close solution.
Were you expecting people to solve this in the space of an interview, or was it intended as a thought process thing?
Evan on January 25, 2008 7:29 AMbillybob,
I don't know about other people.. I think that kind of tasks are not too difficult. It is more to rigorous instead. I don't see a method other than looking at it one by one (by script, of course). The way to speed this up may be with multiple "crawler" or faster connections.
Colin Joss on January 25, 2008 9:12 AMDelmania
You make a good point though I don't agree with your personal opinion as it stands. You state "He (or she) will get the CS or SE degree"
and speaking for myself the answer is not necessarily. However I do read a lot about Computer Science that interests me , I just don't feel the need to validate the learning with a degree.(Similar to not going in for a JEE certification for example). I do agree that some people look for theoretical answers and some for real world experience(I'm with the latter) and this post seems to be with the former.
Aaron G: I would have thought scheduling a phone interview was professional as well. But looking over the 21 interviews I did that year, there were only 5 by phone, not counting recruiters. I think one or two of those were scheduled. To be fair, other than the soccer example I gave, the rest asked if it was a good time to call.
Tim on January 25, 2008 12:25 PM"Umm if the scripting question is soo easy to answer (find phone numbers in 50,000 pages of HTML) could someone explain to me how you would do this"
Use a script (Perl being the obvious choice) that loops through the files, reads in the contents, use pattern matching (regex) to find matching text. Anyone familiar with Perl would consider it trivial.
Steve on January 27, 2008 9:07 AMI noticed a lack of questions about Software Engineering practices like requirements specifications, validation, etc.
Jeffrey on January 27, 2008 10:15 AMI hate to say it, but if someone makes me write some sample code outside the job interview, I will most likely pass. I don't have time to write some stupid sample code for you. There are a lot of other companies out there that are probably just as good as yours and they won't ask me to spend two hours writing a crappy little application so you can see if I can code. Ask me on the interview or don't waste my time.
I generally agree with everything posted here though. I think it's ok to have to google for exact implementation. I don't remember exactly how a radix sort works but I know that it's faster than quicksort for integers in most cases. I also know that a heap works well for priority queues, and I could generally describe what a heap is and why it's useful for that. Could I implement one over the phone? probably not, since I haven't had to do that since college, but it's not like I couldn't figure it out if I needed to. If I'm interviewing someone, that's enough for me in most cases, particularly over the phone. Obviously if I am hiring someone for a particular position that requires a high degree of computer science knowledge, I'd ask more complex questions and expect better answers.
Jeff Tucker on January 28, 2008 1:03 AMThe largest hole that I've personally seen in the process is the incompetent pre-screening that seems to be done by 1) HR people with no concept of either tech vocab or ontology, 2) automated systems parsing for acronyms, or 3) Possibly monkeys throwing darts at a stack of resumes.
I had a cold call phone interview (they called me) from a company in the Chicago area which shall remain nameless. Apparently they got a hold of my resume somehow. After 10 minutes of harassing me about Extreme Programming (which I'd had no experience with) the interviewer got really snotty with me, explaining that there was no place for me there since I didn't work in Extreme Programming. She acted as if I were wasting HER time, despite the fact that SHE called me out of the blue. I asked her where on my resume that it indicated that I was an XP expert, or had ever worked in an XP environment. Well, she got ruder still, and that basically ended the interview.
I later realized the problem. Under my skill set description, I had listed XP as one of the operating systems that I was familiar with. Obviously, whatever mechanism they were using to screen resumes for phone interviews was seeing XP and did not have the ability to put that acronym into context.
In another situation I had an HR person tell me that they needed a person to do RDBMs's. Though when I asked him what that was, he didn't have a clue.
In yet another situation an HR person told me they were looking for a developer with 10 years .net experience. This was in 2003.
Another time, I had an HR person tell me that they needed a developer to do XML, XSL and other EXI technologies. EXI was a proprietary application used by only one company on earth, the company I was currently working for. It was listed on my resume, so he thought he'd just make up a definition to go with it as if it were an open standard or other language.
If companies want to improve there phone screens, start at the front. Don't hire HR folks that can't turn on their computer and don't understand the context in which they are hiring.
As someone already pointed out, it's much more important to be able to quickly adapt, it's not about how much specific coding in any language you can retain, "do you know this and that" - yeah, sure, if I can have access to all my normal references: my previous code I have written and then google, then I can do pretty much everything.
The idea that you need to know how to get a phone number from 50 000 files in an hour without any references to be a good developer is just plain silly. If you are hiring people based on that I feel sorry for you, your company will have hard time adapting when the "new thing" hits and changes everything.
"I want programmers who are good at what they do and are eager to learn to be better at what they do. Google teaches nothing except cutting and pasting."
And being so narrow minded teaches nothing but ignorance. My answer to that is, I would never work for someone like you, you settle for mediocrity and I'm above that.
"good programmers write good code; great programmers steal great code. "
chamberland on January 28, 2008 10:37 AM"good programmers write good code; great programmers steal great code. "
Then who wrote the great code?
KG on January 28, 2008 11:54 AMSteve,
I did massive page searchings last year. It was around 30,000 pages to look for (not 50,000) and definitely not on Perl. I have to leave the 50,000 challenge to Perl experts.
Jeffry,
I think it is because most comment came from non corporate programmers.
Colin Joss on January 28, 2008 12:37 PMJimmy, and other "i have a degree and even I can't answer all those" chaps: I have no degree and no formal education in programming at all and I can. I think one of the schools this process is designed to filter is the school that thinks academic certification is strongly correlated to true programming ability.
dp on January 29, 2008 5:44 AMAfter I finished your test, I'd ask when do I take the personality test, the IQ test, the drug test and the physical...
GH on January 29, 2008 7:12 AMI would love to have had an interview involving those questions! Never happened though...
Jen on January 29, 2008 7:39 AMIf he is saying humans have ten fingers and work in base ten, then a computer should have two fingers, not one, in the analogy.
That depends if you consider "zero" a finger or not, I guess.
Jeff Atwood on January 31, 2008 1:18 AMHi Jeff,
Am pretty sure the Original Joke is:
Why do Programmers get Halloween and Christmas confused ?
dec(25) = oct(31)
:p
Anyways, I would fair pretty well at this interview ... What money would you be expecting in the US for these skills? I'm in NZ and the money ain't good :p
:)
Funny quote: As Steve says, "Computers don't have ten fingers, they have one. So people need to know this stuff."
If he is saying humans have ten fingers and work in base ten, then a computer should have two fingers, not one, in the analogy.
It's sad that his point about learning tricks in other bases is screwed up in the quote he wanted to make memorable :)
Chris Lomont on January 31, 2008 9:37 AMOMFG, that's scary. Some of those questions are easy, but I have no idea what a vector is, for instance. I agree with the first commenter about the difference between a programmer and a math major (well, I'm in college now and with the shipload of math they're making me take, I guess I'll know this stuff eventually, though I'm at a loss at how it'll be useful for general-purpose programming). Also, I had to look up 2^16; I've seen the number before (it's in all sorts of programming guides), but I've never seen where it would be useful information really.
Keith on January 31, 2008 12:27 PMBriefly, the -E flag in the post is redundent to the -a flag. Rsync will likely fail to get crucial operating system files backed up and restored, especially while the OS is live.
Jeff, you intuitively went for a much wiser strategy. There are linux equivalents, but the method given is not one of them.
crap, that was meant for your backup post
ian k on February 1, 2008 1:02 AMSorry to burst your buble, but this kind of interview is why so many companies are full of mediocres, morons and imbeciles. Any monkey on drugs can rebember a few technical solutions. If your company is structured where there is a clear distintion of who is a n analyst, designer and coder, then with this questions you are hiring coders that will putcode before anything else and are incapable of thinking, just on following code specifications written by thinkers.
Last company I worked was full of this kind of people, I call them "Russian Scientist", the ones that will write a solution because is theirs, instead of google one, or buy one cuz is someone else's.
Once we changed our interview process to get the thinking man, the one that understand there is more than one way to skin a cat and that after all we are solving an information problem, not trying to probe 1+1 = 2, the pipeline of issues really cleared out and we had more time to do more coding tht meant something to the bottom line of he company.
Roddy Pineda-Icaza on February 2, 2008 9:22 AMI have never been one to ask low–level questions of an Interviewee. What I usually look for is:
What experience do they have? What types of projects have they worked on? What type of problems have they solved, and how? What environment are they comfortable working in? Will they fit in with our team? Do they exhibit passion for their trade? Do they keep abreast of the latest technologies?
I usually ask a few technology or framework specific questions, however nothing so low–level.
Personally, if I were on an interview and were asked the types of questions you have listed here, I would be deeply offended. That would be a deal–breaker for me. I would leave the interview right there; short of telling the Interviewer to K.M.A. You may think this is a somewhat prima donna reaction. However, I did not spend 14 years building up my reputation, project portfolio, and satisfied client list to be treated like a recent college graduate.
Of course, in reality, I do not have traditional interviews anyway. It's usually a business lunch meeting.
Edward J. Stembler on February 4, 2008 6:06 AMstumbled through, read the article, wanted to say great job, it was a good read, and I'm sure to save it. thanks again!
gabriel on February 5, 2008 7:15 AMAppreciate it. Shaken how I have forgotten my fundamentals. Agreed we don't work on stuff that involves shifting bits. Also agreed that we need to have a strong fundamental base and cannot justify forgetting them, even when you are only studying newer technologies every day. And also agreed that it is impossible to do so and it would take a constant super human effort to do so.
But hey... who said engineering was an easy 9-5 job.
messin' up is fine... not making an effort, not accepting the screw up or trying to justify your failure is what is unacceptable.
Kudos Jeff...!!! I love this one...
Nilotpal on February 6, 2008 2:52 AMBeen interviewing software engineers for 16 years. My experience as an employer has led me recently to the same strategy as Jeff. I now believe and test for:
1. Coding skills with simple loops and a bit of algorithm
- its amazing how many "programmers" can't code even this well
2. Some scripting
- anything really in any script language
3. Design
- User interface or API or OO, anything
4. A passion about something related to work
- a project, a technology, hardware, anything
In my experience, only someone who has all 4 is a really useful developer as opposed to a coder or an architect or an "information guy".
Chris Thorn on February 8, 2008 6:39 AMI had a phone interview with Disney, and I missed nothing - they mostly asked the same questions they had already asked me in written form before the phone screen! But after the interview I never heard a word from them - and they ignored all my emails voice mails. What a bunch of douches.
Snow White on February 10, 2008 8:52 AMThese are smart advices on how to conduct an interview over the phone but I disagree with all the questioning, it is a real torture for the candidate.
I have 7 years experience, and from time to time, I forget things.
Ask me about data structures now, I would be less knowledgeable than i was 3 years ago for instance.
I refuse an interview where i know i will have to answer 50 questions about this and that.
I do not know if you guys do realize that we have left high school ages ago, and it is a real pity and a misery to have to study again, and prepare answers.
What is this exactly ? Do we really need to be super geeks and super intelligent to get a job now ?
I understand why there are so few women in IT when i see how boring the recruitment process can be.
i totally agree with chachaching.
This blog is a joke and i really do not thank you for giving such advices to conduct an interview.
IT interviews are some of the most boring in the market due to geek questions like these.
It is a real torture for the candidate.
Mike on February 11, 2008 5:43 AMI've only been programming professionally for about two years, almost exclusively with .NET.
My first interview was for a junior role, so the questions were more in broad scope with no real coding exercises. I had to take two written tests. One allowed Google, the other didn't. Of course they were watching me from another computer, so it was more of a "will he follow the rules" type of exam.
I've gone through a few phone screenings and they typically ask questions about, JOINS in SQL, what is polymorphism, function overriding/overloading, what is AJAX, etc...
I still find it intriguing that I've never had to provide a code sample. Perhaps it's because the positions I've accepted are not senior in nature and they want to know if I have the potential to do it, and not so much the exact knowledge at this moment of doing it.
I think it all depends. Some people can ascertain from just speaking to someone if they have a clue what they are talking about. Others want to see you actually do it.
I wouldn't be offended if at an interview they asked me to code some of the examples above. I might kind of laugh and say is this COMP 200 Intro to Programming, but I would gladly do it.
Also I almost refuse to believe that someone with over one year of experience cannot code a FizzBuzz example.
Jack on February 15, 2008 9:34 AMI was a music major in college. The only computer class I took had us "turning on the computer" as the first days lesson (easy A class).
I've been indie developing for 1 year, and taught myself everything. Even I can answer half those questions without a second thought. Reversing a string? Are you kidding me? I've written my own scripting for my games because I dislike Lua (even if it did point me in the right direction). Maybe I don't know some theory stuff a CS major would know but it doesn't change the fact that when you get down to it a real PROGRAMMER lives eats and breathes this stuff. A real programmer might blow an interview or two, but when they encounter something they don't know......they will learn it. Plug in those holes, not because they want the job, but because they want to be better.
This isn't to say that there should be a one size fits all interview question. It's stupid for a .Net developer to ask the same questions as a developer who uses Cpp except for general theory to see if they know how to program. At least you guys can get the phone interview. With no CS degree and no "work" experience I can't even get that.
Michael on February 20, 2008 10:43 AMWow, I have to say a lot of that sounds very very familiar. I think that your company contacted me for recruitment. I must say that after three phone interviews and technical screens I decided I wasn't interested.
I can understand the desire to try and determine something of a candidates technical knowledge before bringing them in for interviews. But I was being asked to program, over the phone, very detailed and complex problems. Perhaps since the person doing the phone screen had spent the last 3 or 4 years working on that problem it was easy. But it was a completely inappropriate level of complexity for a phone interview. Also compounded by the problem that I was not being recruited for a programmer job.
I think that I have to agree with chachaching. This may get you a certain type of person and if that is the type of person that you want to recruit then well I guess it works for you. I want to recruit smart, thoughtful people, who have a broad and diverse background, and know how to research and solve problems. I really care nothing for the ability to regurgitate some meaningless program code at the drop of the hat.
steve on February 27, 2008 4:47 AMI've been asked these sort of questions. If they're prefixed by "I've read your application and I'm sorry to have to ask these but we have to treat all applicant identically" then I'll humour them. Please do excuse me if I take a moment to remember what a Fibonacci number is, as they are not something I've ever had cause to actually code here in the Real World.
But if the employer's representative hasn't looked at the 40KLOC off FOSS software I mentioned in my application, then it's time close the discussion, as I've just learned all I need to know about the prospective manager's own skills and willingness to work hard.
gdt on February 27, 2008 6:12 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |