Most tech interviews still test the part of the job AI does well enough (I wouldn't say best).
The code is the obvious part.
But it's not the only one.
1. What's Happening
This is what we, software engineers, do when we approach a new problem at work, with or without AI:
- Get a problem (a task).
- Clarify open questions.
- Analyse it.
- Think about possible solutions. Weigh different options.
- Choose one, based on trade-offs. Hopefully not on gut feeling.
- Implement.
- Test.
- See if we can do better. If yes, re-implement.
- Test.
- Release.
For years, interviews went deep on step 6. The code.
Agents now write, plan and refactor code fast. Agents can't tell you whether you're solving the right problem, or whether you are solving it the right way.
What if we give the candidate a vague business problem, the way a stakeholder would say it, and watch how they work through it.
denstuk's example: Support gets about 2,000 calls a month asking "Where is my order?".
2. Which Steps the AI Takes Away
It's not only step 6.
AI takes steps 4 and 5 too. Looking for possible solutions is exploring options: the algorithms, the logic, the "here are three ways we could do this". That's exactly what AI does well enough now. Ask it to give you the code to solve a problem, and it prints it out for you in less than a minute.
But it gives the candidate one option. The candidate scans through it, most likely agrees with it, and asks the AI for some adjustments.
When I ask the candidates to think of a different way, a different logic, to solve the same problem, they struggle. As any normal human being would. Psychology calls this effect priming. When we are shown one possible way forward, we struggle a lot with taking a few steps back and thinking about another way to go about the same problem.
I am not saying that AI is not good. In September 2025, at the ICPC World Finals, the world's biggest university programming contest, OpenAI's system solved all 12 problems within the five-hour limit. The best human team solved 11. Google DeepMind's Gemini solved 10, including one no human team solved.
What I am saying is that we are still hiring, hopefully forever, humans to do technical work for us, AI or no AI. We need these humans to keep their ability of independent thought and problem solving. These are the two strongest traits of the best employees, after all.
So an interview built around algorithm puzzles is testing the part where AI already beats the best students in the world. And priming the human behind it to think exactly like the AI.
I would argue AI also takes away step 3. I have caught myself thinking "I am not an expert in this field, so I will follow the AI's recommendation". But when this happened, I gave the AI full control over my work, and I trusted that it would do its best for my professional benefit. It has shown me multiple times that it cannot. Not unless I give it very clear, very specific instructions. And the only way I am capable of giving it these instructions is because I have done the job myself, successfully might I add, in the past. Several times. With proven results.
3. The Catch
There's a trap here, and I believe you all see it.
If AI does steps 3 through 6, people stop practising them.
Anthropic tested this on engineers. In a randomised trial published in January 2026, 52 mostly junior software engineers learned a new Python library, half of them with an AI assistant. Afterwards, the AI group scored 50% on a quiz about what they'd just used. The group that coded by hand scored 67%.
The biggest gap was in debugging: spotting and fixing code that's wrong. Which is exactly what we need employees doing when using AI at work.
It's a small study, and the quiz came straight after the task.
But one finding matters for interviews: how people used AI changed the result. Those who asked it to explain, and used it to understand, scored much better than those who just let it write the code.
So I wouldn't drop fundamentals from interviews. If I am to stop asking candidates to produce the algorithm, I will start asking them to judge one.
What would you check before you trust it?
Why this option and not that one?
What breaks first?
4. A Smart Little Task, and a Question for denstuk (Who's on Substack Too)
Give candidates a support ticket that says some users aren't getting a confirmation email, and nothing else. Some candidates start listing fixes straight away. Others first ask whether it's all users, whether the email service is a third party and what the logs say.
You learn a lot from that in a few minutes. It's the missing steps, live.
But let me ask you the question I always ask: how do you score it, so two interviewers reach the same answer?
Some parts are easy enough to check: the right questions, the way of thinking, a good solution.
The rest is "less measurable and mostly based on gut feeling." Mmm, really bad if our goal is to conduct unbiased structured interviews.
Judgement is harder to score than code, so it needs more structure, not less.
You already know the drill.
Same problem for every candidate. A written scoring guide, agreed before the first interview. Notes taken during the interview, not from memory afterwards.
Otherwise "how they think" quietly turns back into "whether I liked them".
5. How to Interview for the Left Side of the Loop
1. Start with a symptom, not a system. "Some users aren't getting a confirmation email" beats "Design an email service". Let them find the real problem.
2. Score the questions they ask. In the first ten minutes, write down every clarifying question. That's the missing steps, and it's most of the job now.
3. Give them the options. Hand over three solutions to the same problem, the kind AI would suggest. Ask which one they'd choose for your team, and why. That's step 4, or 8.
4. Ask what they'd check. Before they'd trust the chosen option, what would they test? This is where you find out if they really understand the algorithm and the logic, without asking them to write it from memory.
5. Write the scoring guide before the first interview. What does a strong answer look like at each step? If two interviewers can't agree on it, you're scoring a feeling.
6. Keep one short look at the code. Implementation isn't gone. Ask them to read a small piece of code and say what's wrong with it. Reading and debugging is the part AI users lose first.
6. Your Checklist
- Our interview starts with a vague problem, not a well-known product
- We score the clarifying questions, not only the final answer
- We ask candidates to choose between options, not to produce them
- We test whether they can spot what's wrong, not whether they can write it
- We have a written scoring guide for problem-space answers
- Two interviewers would give the same candidate the same score
- No step of our interview is an algorithm puzzle AI solves in seconds
TL;DR: AI took the code, and the options too. Interview for the steps it can't take: understanding the problem, asking the right questions and choosing. Then score them properly.
Which step of the loop does your interview still spend most of its time on?
Want Help Interviewing for Judgement?
If you'd like your interviewers to score clarifying questions and choices as consistently as they score code, our Tech Interviewer Training teaches structure, scoring and feedback. And if you'd rather have an experienced engineer run the round, that's what our Tech Assessments are.
Tech Interviewer Training
Teach your team structure, scoring and feedback, so every round measures what it says it does.
Tech Assessments
Structured technical interviews run by a senior engineer or CTO from outside your team, with a report within 24 hours.
Is Interview as a Service Right for You?
Not sure where to start? Our free Buyer's Guide to Interview as a Service explains when handing your technical interviews to a partner makes sense, and when it doesn't.
Sources
- denstuk, Hiring Engineers When Agents Write the Code. 9 November 2026, and denstuk's reply to Kat's comment, 11 November 2026.
- Maximilian Schreiner, OpenAI outperforms humans and Google at the world's top collegiate programming contest. The Decoder, 17 September 2025.
- Judy Hanwen Shen and Alex Tamkin, How AI assistance impacts the formation of coding skills. Anthropic, 29 January 2026.
This post first appeared in Shortlisted by Kat. Photo by Dan Freeman on Unsplash.



