What you know got you hired in 2024. How you think will get you hired now.
One of the most AI-heavy companies in the world found exactly this when it read its own interview notes.
To be honest, I was hiring for how people think long before 2024. It has been the main focus of my interviews since 2018.
What Anthropic Found
Lars Schmidt, who works on talent at Anthropic, shared a new report on LinkedIn last week. Anthropic studied how AI changed work and hiring inside its own company. The report is by Nathan Wilmers, Regina Catipon, Maxim Massenkoff and Peter McCrory.
Three findings stood out to Lars, and to me.
AI agents now do most of the hours. By August, hours worked by AI agents outnumbered human hours six to one.
There are fewer juniors. The share of junior-level people in technical departments fell from 21% to 14% since early 2025. The share of junior technical hires fell even faster, from 27% to 9%.
Interviewers look for different things now. Anthropic used an automated tool to read hiring managers' interview scorecards for the thousands of people it has hired since 2024. In 2024, about 5% of technical assessments mentioned the candidate's track record with AI tools. By 2026, that share had tripled. For technical candidates, the biggest increase was in comments on autonomy and judgment. Comments on domain expertise shrank.
One manager put it like this: "I expect models to continue to be jagged, so for my team I want them to be fungible across domains."
In plain words, the AI can do a lot of the implementation. What's left for people is deciding what to do, and doing it without someone holding their hand.
My own translation is simple. If you can't do it without AI, you won't be able to do it with AI helping you at every step.
Before You Copy Anthropic
The report itself says to be careful, and I agree.
Anthropic is not a typical company. It builds the AI, its people use it more than almost anyone, and it tripled in size in the last year. The authors say plainly that many of its practices "cannot be replicated across the economy". The scorecard numbers also count what interviewers wrote about, coded by a machine, so they don't tell you how anyone was scored.
"Creative use of AI tools" isn't defined anywhere either. If you add it to your scorecard tomorrow, every interviewer will read it differently. That's exactly how bias gets in.
So take the direction from the report and leave the wording. Ask less about what candidates know, and more about how they think and decide.
What This Looks Like in My Interviews
I've been testing for judgment for years. I never believed in memory questions, long before AI made them easy to answer.
At university, I had a programming exam where we wrote a program on a piece of paper. There was no Googling and no documentation, and nothing to check the syntax against. I thought it was an absurd way to test someone then, and I still think so today.
One question I use goes like this: "A user adds items to their cart, enters their card details and hits submit. Some users aren't getting a confirmation email. Here's the ticket. What do you do?"
Some candidates start listing debugging steps straight away. They know how to debug.
Others first ask questions. Is it all users or only some? Is the email service a third party? What do the logs say?
They define the problem before they solve it. That second group is what Anthropic's interviewers now describe as autonomy and judgment. No AI tool can ask those questions for you if you don't know they matter.
In our live challenge, candidates can ask me questions and look things up, as they would at work. One of the things we note is what kind of help they needed. That tells me a lot about autonomy. Someone who asks one sharp question and runs with it is very different from someone who needs every step explained.
Looking up the syntax of a library function isn't a weakness either. It's a 5–10 second search and simply part of the job, so nobody loses points for it.
Three Things to Change in Your Next Interview
- Give a problem with a gap in it, and score the questions. A clean, fully specified task tests recall. A ticket with something missing shows whether the person asks before they build. Give every candidate the same gap. Before the first interview, list the 2–3 (or 5–6) questions a strong candidate should ask, and mark how many of them each candidate thought of.
- Ask "why" and note the help they needed. Why this approach and not another one? Why did you accept that AI output? People with judgment can explain their choices, and people who copied can't. During a live task, note what they asked and when. One good question at the right moment shows autonomy. Asking for the next step every two minutes shows the opposite.
- Define good AI use before you score it. "Creative use of AI" means nothing until you write down what good looks like. Maybe it's knowing when not to use AI. Maybe it's checking the output before trusting it. Decide, tell the candidate the rule, and score it the same way for everyone.
This is an afternoon of work. Possibly less.
Don't Stop Interviewing Juniors
If judgment matters more than what someone knows today, a junior who learns fast can beat a senior who stopped learning. Ask juniors how they learned something new recently, and how they checked they got it right.
A Quick Checklist
- At least one task in our process has a gap on purpose.
- Our scoring guide rates the questions candidates ask.
- Every answer gets a "why" follow-up.
- We've written down what good AI use means for this role.
- Candidates hear our AI rule before the task starts.
- Interviewers note what help the candidate needed.
- We still interview juniors, and ask how they learn.
So, what does judgment look like in the interviews you run, and how do you score it?
Sources
- Lars Schmidt (Anthropic), LinkedIn post. 17 October 2026.
- Wilmers, Catipon, Massenkoff and McCrory, How Employment Is Changing at Anthropic (PDF). Anthropic, 16 October 2026. Web version.
This post first appeared in Shortlisted by Kat. Photo by Jens Lelie on Unsplash.
Interviews That Test Judgment
Every AssessDev tech assessment includes questions that test how candidates define a problem, and a live challenge where candidates can ask questions. We score how they reason.
