It’s popular to be critical of live-coding interviews. But like many issues that people reduce to “X is good” or “X is bad”, I think the question of live coding interviews is actually more complicated than that. Here’s my take.
The criticisms of live coding interviews
The content isn’t realistic
This can obviously be true. If a candidate is asked in an interview to, for example, live-code an implementation of a linked list or a binary search, then the candidate is being asked to do something that’s not really representative of what they’d actually do in a typical programming job.
Programmers who pass these tests are demonstrating that they’re capable of passing computer-sciencey type tests, but they’re not necessarily demonstrating that they have skills that match the job they’re interviewing for. So these types of tests aren’t very smart ones.
But to use these as a criticism of live coding tests is a confusion. If many live-coding tests contain unrealistic content, that doesn’t prove that live-coding tests are bad, it only proves that live-coding tests with unrealistic content are bad.
The spotlight and pressure changes the outcome
Sometimes developers choke on live-coding exercises not because they’re not capable of completing the task but because the interview process made them too nervous to think clearly enough.
I do think this is a valid criticism of live-coding tests. Having said that, there is at least one thing that can be done to mitigate the nervousness factor. That is: don’t make it a race. A live-coding test doesn’t have to be “complete X in Y amount of time or you fail”. When I’ve run live-coding tests myself in the past, I limit the time we spend, but I don’t require candidates to finish the exercise as a condition to passing. Rather than “Did you complete the entire exercise?” the question is “Did you get anything to work at all?” Candidates who fail to get anything to work don’t pass the test.
Why I’m in favor of live-coding interviews
End products don’t tell the whole story
Given a particular challenge, two different developers may arrive at roughly the same solution. But one developer may have struggled with the challenge for two hours while the other developer solved it in five minutes. I contend that this difference matters.
If I give developers a take-home exercise, then all I can see is the end product. I don’t get to see how they arrived at the end product. Maybe developer A has 20 years of experience but they approached the task in a slapdash and thoughtless manner and struggled for hours before finally piecing together the right answers. Maybe developer B has only 3 years of experience but methodically homed in on the answer in just a few minutes because they’re smarter.
Selecting for intellectual character over experience
Because end products don’t tell the whole story, judging by end products alone may select for experience over raw brainpower. If the candidate submits a solution that works but looks sloppy, how do you know if that mistake is due to lack of intelligence or just lack of experience?
It would be a real shame IMO to reject someone who’s really smart and has great potential just because at the current moment they’re not very experienced. I’d much rather take an inexperienced but smart person who will be 10X better a year from now than a very experienced but not-very-smart person who is not only slow but also will never get much better.
And as a side note, I think “smart” is way too vague of a word to use when evaluating programmer potential. Plenty of people are “smart” in the sense that they can solve hard logic puzzles or write complicated algorithms, but many of those “smart” people also make shockingly incompetent developers because their intelligence is of a very narrow kind. Other traits that I think make a good programmer include things like intellectual honesty, intellectual rigor, inquisitiveness and discipline. These qualities make a world of difference. And I don’t think they’re very evident in the end result of a take-home coding test.
When scrutiny isn’t necessary
The only reason to do something like a live-coding interview, or any interview process at all, is if you don’t yet know how good the person is and if they’re the kind of person you’d like to work with. If other, better ways are available, then certain aspects of the interview process becomes less necessary. For example, if someone has a large body of publicly available work, or if you’ve already worked with them before, then you can potentially use that to judge the person’s abilities and you don’t have to contrive tests for them. But most programming candidates don’t have much to be judged by, and so unfortunately in most cases we have to do this painful song and dance we call the interview process.
Summary
I think live-coding tests can either be done well or poorly. Live-coding tests have some unavoidable drawbacks, but so does everything. The value of live-coding tests, when done in a smart way, is that they reveal how a person thinks and behaves while coding. To me this is the most important thing to learn about a programming candidate. I don’t see how this information can be gained any way other than actually watching them code.
Do you need to watch how car is made before buying it? Do you ask doctor for a trial before getting your health fixed? Do you have anyone looking at your paper while writing a school test? Then why aren’t you interested in end product? Isn’t it what really counts? Live interviews assess anxiety, not software skills – there is research on it, please have a read.
„Maybe developer B has only 3 years of experience but methodically homed in on the answer in just a few minutes because they’re smarter.”
Or maybe they just slept well that particular night and simply had a better day. There is many more important factors to impact one-time assessment than someone being „smarter”. Wake up, life is not a binary code.
I once had a timed pre-screening survey via email. I told my recruiter when I wanted to take it and they scheduled an email to be delivered at that time. This established when a take-home code screening begins. And the rest is in the commits – yes, you provide the candidate with a repo. This not only tells you have fast they work, but lets you evaluate their commit hygiene – an important skill that I’ve never seen addressed in an interview. Finally, you have an in-person post-mortem review of their work, in which the interviewer has the opportunity to go over the candidates thought process as they review their commits.
This is a much more realistic setting in which to evaluate a candidate’s capabilities. Leave them alone, and let them do their thing. Just like in the office.