Watch an engineer solve a problem, and you may see less than half of what they can do.
That is the finding of a randomised trial from NC State and Microsoft. Mahnaz Behroozi and her co-authors gave 48 computer science students the same whiteboard problems. Half solved them alone in a private room. The other half solved them in front of an interviewer.
The watched group's performance dropped by more than half. Their stress and cognitive load were measurably higher. The only thing that changed was the audience.
Tech hiring treats live coding as its most realistic test. Psychology says being watched is noise in the measurement. Both can be true at once, and that's worth sitting with.
So I went through the research behind every coding test format I know and ranked them.
Why Being Watched Hurts
Psychologists call it social facilitation. Having other people around speeds up simple, well-practised tasks. It slows down complex and unfamiliar ones and makes them worse. Bond and Titus showed the pattern across 241 studies in 1983.
A new coding problem is a complex, unfamiliar task. Your working memory is busy holding the problem. When someone is watching, part of it is busy watching them back.
The Ranking
No study has compared coding test formats head to head. So I placed each format in the research category it belongs to. Then I scored it on five things. Those are how well it predicts job performance, how much stress being watched adds, how structured the scoring is, how candidates feel about it and whether you can trust who did the work.
The validity numbers come from Sackett and colleagues' 2022 re-analysis. A validity of .42 means the method tracks later job performance much better than one of .19. I wrote about that paper in Cognitive Ability Is Not the Best Predictor of Job Performance.
| Format | Why it ranks here | What psychology says |
|---|---|---|
| 1. Structured technical interview | Every candidate gets the same job-relevant questions. Answers are scored against a rubric agreed in advance. It feels like a conversation. | The best single predictor of job performance (.42). Job knowledge tests come close (.40). Candidates rate interviews more favourably than any other method. |
| 2. Private work sample, then a walkthrough | A realistic task done alone for 20–30 minutes. The candidate then explains what they did and why. | Work samples predict at .33. Working alone removes the stress of being watched. The stress study's authors recommend exactly this. |
| 3. Collaborative live task | A realistic problem with the interviewer present. The candidate can ask questions and search. | The same work sample signal. Being watched still slows complex work. Questions and search take some of the pressure off. |
| 4. Take-home with a rubric and a review call | Realistic and private. It costs candidates unpaid evenings. Without the review call you can't tell who wrote the code. | Work sample validity, as long as you can trust the work. Drop the call and it belongs at number 6. |
| 5. Timed online coding test | Scored automatically and the same for everyone. The problems are rarely job-like, and nobody sees who solves them. | Standard scoring helps. Candidates rate test-style methods well below interviews and work samples. |
| 6. Live algorithm puzzle, watched | A new, hard problem under time pressure with an audience. | These are the conditions where people under observation perform worst. |
| 7. Whiteboard coding | Everything in row 6, plus no compiler and no autocomplete. | In a randomised trial, being watched cut performance by more than half. |
| 8. Unstructured technical chat | "Tell me about your experience." It feels informative. | Unstructured interviews predict at .19. Interviewers felt just as confident after random answers as after real ones. |
| 9. Brainteasers | "How many golf balls fit in a plane?" | There is no evidence they predict anything. People who favour them score higher on narcissism and sadism. |
Why We Use Number 1 and Number 3
At AssessDev, every technical interview has two parts. First come 10–12 structured questions for the role. Then there's a live challenge of about 30 minutes. Candidates can ask questions and use search, but no AI tools. You can see the shape in our sample interview structure.
A table that ranks the private work sample above our live challenge is awkward for me to publish. So here is why we still chose it.
Two measures beat one. The structured questions tell us what someone knows and how they reason about it. The challenge shows us what they do with a real problem. They measure different things, and both rank near the top of the validity list.
Asking questions is part of the job. Real engineering tickets are vague. I want to see whether a candidate clarifies the problem before writing code. A private task can't show me that, and it matters more now that AI writes so much of the code.
We see the work happen. There's no doubt about who or what wrote the solution. Candidates don't lose an unpaid evening to a take-home. Our clients' engineers don't lose hours reviewing one.
The candidate leaves with an answer. When someone gets stuck, we tell them the answer. That only works when we're in the room together.
The cost is real, though. Being watched takes something away from the candidate's performance. Our job is to take away as little as possible.
How to Help a Candidate Relax During a Live Challenge
Most of these take seconds. All of them come from research on stress and performance.
- Tell them the format in advance. Share how long the challenge takes, what tools they can use and that questions are welcome. Surprises use up the same working memory they need for the problem.
- Start with the conversation. Leave the challenge until rapport is built. A cold open is the most stressful version of the task.
- Say that nerves help. Students who were told that arousal improves performance scored higher on a real maths exam (Jamieson and colleagues, 2010). One sentence such as "Feeling a bit wired is normal, and it actually helps" does the job.
- Give them a minute to write things down. Anxious students who spent 10 minutes writing about their worries before an exam averaged a B+. Those who didn't averaged a B– (Ramirez and Beilock, 2011). Let candidates jot down their first thoughts before anyone talks.
- Give them quiet time. Mute yourself and turn your camera off for a few minutes. Then ask them to walk you through their approach. This is the fix the stress study's authors propose.
- Don't narrate their typing. Silence while they think is fine. A comment on every keystroke is an audience they can't ignore.
- Hint, then answer. If they're stuck, offer a hint. If they're still stuck, tell them the answer at the end so they leave having learned something.
There are more ideas in how to help a nervous candidate relax.
What This Doesn't Mean
The stress study tested 48 students. Experienced engineers may cope better. A senior engineer who has given dozens of talks may not lose half their performance.
The ranking is also my own mapping of each format onto broader research categories. A brilliant take-home with a thoughtful review call can beat a sloppy structured interview.
And a .42 is still modest. The best tool on the list is the best of a weak set.
Three Things to Do on Monday
- Delete the brainteasers and the whiteboard. Nothing in the evidence argues for keeping them.
- Write the rubric for your live task before the next candidate sees it. Without one, your challenge is an unstructured chat with a keyboard. Design a live coding task in 4 simple steps shows how.
- Add five quiet minutes to your live challenge, then a walkthrough. It costs you nothing and gives you a truer picture of the candidate.
So, which row is your coding test in?
Sources
- Behroozi, M., Shirolkar, S., Barik, T. & Parnin, C. (2020). Does Stress Impact Technical Interview Performance? ESEC/FSE 2020.
- Bond, C. F. & Titus, L. J. (1983). Social facilitation: A meta-analysis of 241 studies. Psychological Bulletin, 94(2), 265–292. Summarised in Principles of Social Psychology.
- Sackett, P. R., Zhang, C., Berry, C. M. & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(11), 2040–2068.
- Hausknecht, J. P., Day, D. V. & Thomas, S. C. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57(3), 639–683.
- Dana, J., Dawes, R. & Peterson, N. (2013). Belief in the unstructured interview: The persistence of an illusion. Judgment and Decision Making, 8(5), 512–520.
- Highhouse, S., Nye, C. D. & Zhang, D. C. (2019). Dark motives and elective use of brainteaser interview questions. Applied Psychology.
- Jamieson, J. P., Mendes, W. B., Blackstock, E. & Schmader, T. (2010). Turning the knots in your stomach into bows: Reappraising arousal improves performance on the GRE. Journal of Experimental Social Psychology, 46(1), 208–212.
- Ramirez, G. & Beilock, S. L. (2011). Writing about testing worries boosts exam performance in the classroom. Science, 331, 211–213. Reported by UChicago News.
Photos by rupixen, Nubelson Fernandes and Christina @ wocintechchat.com on Unsplash.
Structured Questions and a Live Challenge, Done for You
Every AssessDev tech assessment pairs structured questions with a live challenge, scored against a rubric your team agrees with us.
