Anyone can tell you which AI tools they use. That's neither an AI skill, nor something we should be evaluating.

The skill I care most about is knowing when the answer is wrong. And it's the one thing most interviews never test.

1. What's happening

AI skills are in almost every job ad now. In TestGorilla's 2026 survey of 1,928 senior hiring leaders in the US and UK, 95% said AI competency is a stated hiring requirement.

So interviews are changing too. In July, Emma Jacobs wrote in the Financial Times (republished by The Irish Times) about job interviews becoming AI tests. McKinsey added an AI part to its interviews for junior consultants. Michael Page asks candidates to improve a weak prompt. A startup founder gave a marketing task and watched a candidate build a landing page with Claude Code.

Still, McKinsey's Marie Christine Padberg said:

We don't test how to prompt or how many tools they know.

That's the right instinct.

Unfortunately, most teams are doing the opposite.

2. The one thing teams get wrong

They test whether candidates know about AI, not whether they have been using it on a daily basis. Forget about catching it being wrong altogether.

The same TestGorilla survey found that:

  • 37% set their minimum bar for AI fluency at tool awareness: knowing the tools exist.
  • 19% leave AI assessment entirely to each hiring manager. (great, but how is this evaluated?)
  • Only 26% ask candidates to show they can use AI on their own.
  • 31% of hiring managers said their main problem is telling real understanding apart from fluent vocabulary.

TestGorilla's CEO, Wouter Durville, put it simply: a candidate can describe a workflow convincingly "without ever having built one."

Our job as interviewers is to be able to spot the difference between fluent vocabulary and actual knowledge.

On my team, we go beyond that. We test whether the person knows how to do the job well without AI. We then evaluate how they would instruct the AI to do what they already know how to do.

59% of these organisations said they'd made a bad AI hire in the past year. Still, the survey doesn't prove the low bar caused those hires. TestGorilla also sells skills tests, so read it with that in mind. But a bar that low lets almost anyone through.

3. Why asking doesn't work

"How good are you with AI?" sounds like a fair question.

But let's pause here. As a human being, if you were asked today to describe how good you are at something, what would you do?

My bet is that your initial reaction will be a surprise. After all, during interviews, we've ditched these types of questions long ago. No one asked you how good you are at writing code in Python. You were usually given a short problem to solve and told to code the solution. That's very different from asking you to give a talk, or write an essay, on how good you think you are at writing code in Python in general.

Following this line of thought, it baffles me that we think "How good are you with AI?" can actually give us any valuable signal we can rely on when making a decision about a candidate.

But the story unfolds even further.

People who use AI think they did better than they did. In a study led by Aalto University, published in Computers in Human Behavior in October 2025, about 500 people solved logical reasoning questions from the LSAT, half of them with ChatGPT. The people using AI overestimated how well they did.

The ones who felt most AI-literate were the most overconfident. "What's really surprising is that higher AI literacy brings more overconfidence," said Professor Robin Welsch. Most people asked ChatGPT once per question, and often took the answer without checking it.

So the candidate who sounds most sure about AI may be the one who checks it least. Which for anyone hiring should be a red flag. One of the brightest.

And "almost right" is the real problem. In the Stack Overflow 2025 Developer Survey, 84% of developers said they use or plan to use AI tools. Only 33% trust its accuracy, and 46% don't. The biggest frustration, named by 66%:

AI answers that are "almost right, but not quite."

Almost right is exactly what a confident, fluent candidate can't see. It's exactly what you need them to catch. It's what helps you make the decision on whether to move that candidate onto the next stage of the interview process.

4. A quick story

I studied AI long before it was in every job ad. Before it was cool. My master's degree was in Natural Language Processing, from 2010 to 2012.

What is Natural Language Processing, you ask? It is the part of AI that works with text (because finding the fastest route on a map for a car driving back home in rush hour does not need any understanding of human languages and natural speech), so in today's context: everything related to LLMs (ChatGPT, Claude, Gemini, etc.).

Two weeks ago, I had to fail a candidate. He sounded great at the interview, when we were at the "How good are you with AI?" stage. Then, as usual, I asked him to show me what he generated with AI (Claude in that particular case). He opened Visual Studio, his IDE of choice, and showed me well-structured code written in Java. Everything looked perfect, which earned him some points. After all, I have seen the most unorganised, poorly written pieces of code written by an LLM during the past 2 years. He showed me he knew how to avoid that.

But then we ran the code. I do not trust it working fine unless it is proven to me. The output was completely wrong. It was supposed to give us a number after some aggregations and calculations made from real-life data, and the result printed in front of my eyes was almost twice as big as the number we expected to see.

I spotted it immediately. It burned holes in my eyes. The candidate did not, and seemed quite content with his interview performance.

In my own interviews, most of the time, I ask candidates not to use AI, and I tell them why: it primes their thinking before I've heard it. Looking things up in the live challenge is fine, like at work. What I want to see is how they think, and whether they can solve the problem without AI, because that's the skill they'll need when the AI gets it almost right. Or wrong, like what happened two weeks ago.

5. How to test it instead

1. Stop asking which tools they use. A list of tools tells you what they've heard of, and it'll be out of date by next year.

2. Give them AI output to review. Hand over an answer, a piece of code or a summary an AI produced, with a real mistake in it. Ask: "Would you ship this? What would you check first?"

Sometimes I give them an AI-generated piece of code which returns the wrong output. Sometimes, I give them something that is almost right, and which needs a bit more digging to force the code to return the wrong output.

3. Ask about the last time AI got it wrong. "Tell me about a time an AI answer looked right and wasn't. How did you find out?" Real stories have details: the task, the mistake, what they did next.

These real stories are difficult to score though, so if you include such a question in your interviews, make sure you have prepared how to evaluate it.

4. Score the checking, not the prompt. A clever prompt is nice. Noticing the missing edge case, the made-up source or the wrong number is the job.

5. Write down what a good answer looks like first. Before the first candidate, agree what a strong review sounds like. Then every candidate gets the same task and the same scoring guide, so it doesn't come down to one manager's taste.

6. Make "I wouldn't use AI for this part" a good answer. Knowing when not to use it is part of the skill.

6. Your checklist

  • We've written down what "AI skills" means for this role
  • Our minimum bar is more than knowing the tools exist
  • Candidates review real AI output with a real mistake in it
  • We ask about a time AI got it wrong, and listen for details
  • We score how they check, not how they prompt
  • Every candidate gets the same task and the same scoring guide
  • "I'd do this part without AI" can earn points

TL;DR: Most teams test whether candidates know the AI tools. Test whether they can tell when the AI is wrong.

What's the best question you've heard for testing someone's AI skills?

Want Help Testing AI Skills Properly?

If you'd like your interviewers to test how candidates check AI output, not how they talk about it, our Tech Interviewer Training teaches structure, scoring and feedback. And if you'd rather have an experienced engineer run the round, that's what our Tech Assessments are.

Tech Interviewer Training

Teach your team structure, scoring and feedback, so every round measures what it says it does.

Tech Assessments

Structured technical interviews run by a senior engineer or CTO from outside your team, with a report within 24 hours.

Is Interview as a Service Right for You?

Not sure where to start? Our free Buyer's Guide to Interview as a Service explains when handing your technical interviews to a partner makes sense, and when it doesn't.

Sources

This post first appeared in Shortlisted by Kat. Photo by Markus Winkler on Unsplash.