Almost every hiring team says it runs structured interviews. Ask them to show you exactly how they verify this, and it gets quieter.

The recordings are there. The transcripts are there.

Imagine listening to the noise of crickets in a quiet wheat field at dusk right now.

That's what the recordings hear. Silence. Nobody has opened them.

This is what a LinkedIn thread with Hung Lee (he is on Substack too if you want to check his content) taught me about the proof.

1. What's Happening

Last week Hung Lee asked a simple question on LinkedIn: Structured interviewing… does your company do it, really?

He started with what everyone agrees on. Same questions, same order, and every answer scored against criteria you agreed before the interview. Fairer for the candidate, clearer for the hiring manager.

Then he shared a finding from research by Textio and Kyle & Co. According to Hung, 81% of the organisations had an interview intelligence tool, the kind that records and transcribes your calls. But a much smaller share reviewed what it collected in any systematic way.

His plain-language version: "we're recording and transcribing the zoom calls but nobody is reading the transcripts".

He ended with a two-question poll. Do you have an interview intelligence tool?

And do you review what it gives you?

I think the second question is the one most teams would rather skip.

2. What Everyone Else Said

The comments were some of the best I've read on this topic in a while. Here are the ones that stayed with me.

It's an ownership problem. Christopher Weis, a global talent acquisition leader, put it in one line: "The gap is ownership." Recording happens on its own. Reviewing is a choice. Without someone accountable, he said, the tool becomes an expensive archive.

If you ask me, that's a very expensive archive just to wait to be deleted in a few years.

It's a gym membership. John Fuller called it "a gym membership with a nicer dashboard." Someone has to own a weekly look at a few real transcripts.

A quick question for you here: have you ever bought a monthly gym membership and have you, during that particular month, gone exactly zero times to the gym? I have not, but I don't like the gym, so that's that. You'll find me in a yoga studio instead, and there, I don't skip.

Start by counting. Mayur Macwan, founder of RoundOne AI, said the cheapest use of the transcripts is just counting whether the agreed questions were asked. No AI needed. His bet: plenty of "structured" interviews drift off script after question two, and nobody knows.

Ask the panel. Alison Hemmings shared the easiest test of all. Before the debrief, ask everyone on the panel for the top three things they were assessing. "If the answers don't match, it was never structured."

Check the scores against the hires. Ahmed Ismail Azzabi, founder of CodeDuel, said the same gap shows up after the interview: nobody checks whether the scores predicted anything six months later. Ernest Lawlor made a similar point from his time auditing trainers against a checklist. Following the process and the process working, he said, are "two different questions."

Keep room for nuance. Ann Bender Dalhäll warned against oversimplifying. Hiring a warehouse operative and hiring a CEO need different amounts of structure. Her view: as much structure as possible, with room to dig deeper when it matters.

And one comment pointed at AI. Mark Dawkins, a technical talent partner, linked a new study on AI interviewers. His reading of it: the gains came from standardisation. The AI followed the interview guide every time, and the humans drifted. So the value isn't the recording, he said. It's checking that the structure was followed.

Hung then asked Mark the question I'd ask too: why do humans drift?

3. What the Study Found

I read the study Mark shared before quoting it.

AI interviewers led to more job offers. In a field experiment by Brian Jabarian and Luca Henkel, about 70,000 applicants for real jobs were randomly given either a human recruiter or an AI voice agent for their interview. Human recruiters still reviewed the interviews and made every hiring decision. Applicants interviewed by AI were 12% more likely to get a job offer.

Remember this one already? I had mentioned it trice, I believe trice, so far. The caveat here is that the applicants are entry-level customer support people in the Philippines so take its findings with a grain of salt.

The authors say those gains carried through to more people starting the job and staying, with no drop in the productivity of the people hired. And they explain it with one word I care about a lot: structure. The AI interviews were more structured and consistent, while still responding to each applicant.

The abstract doesn't measure human "drift" directly. That part is Mark's reading, and a fair one.

But the lesson isn't "let AI run your interviews". It's that consistency is worth something you can measure.

And human interviewers can have it too, if someone checks.

Trust me on that one.

4. My Answer to Hung, and the 84 Characters

My first comment on the post was short. The review gets easier when it's small and regular. Having run more than 2,000 technical interviews myself, the check I'd start with is one question a week: compare how three interviewers scored the same answer. Gaps in scoring show up fast, and they tell you exactly where your criteria need sharper wording.

Hung liked the technique, then raised the real problem. Who is the reviewer, and why would they bother? In his words, there's no immediate gain for the reviewer, and there may be "a social and political cost". Nobody wins points for telling a senior hiring manager they went off script.

That's a good question. And for the first time in forever, my answer didn't fit in a LinkedIn comment.

Kat's LinkedIn reply to Hung Lee, with the warning 'Exceeded the maximum character limit.' and -84

LinkedIn told me I was 84 characters over the limit. So I cut it down and posted the shorter version. Here's the full thinking behind it.

The reviewer checks the structure, not the people. Their job is to confirm that interviewers and hiring managers follow the structure that was agreed before the first interview.

Our tool makes the first check easy. We use an interview note-taker app in our interviews. Everyone interviewing for the same role at the same employer asks the exact same questions, and we can see which ones were skipped. They give the same live challenge and score it with the same rubric. Some of our clients do something similar on their own platforms.

Scores mean something because we trained for them. Our interviewers score each answer from 1 to 5. Before they interview anyone, they're trained on what a 1, 2, 3, 4 and 5 look like, so they can score during the interview, not from memory afterwards.

And yes, we still calibrate. From time to time we check the scores against each other, because scoring is where human emotions, bias and everything else we want out of the interview can slide right back in.

I couldn't answer Hung's question for every company. But a live example of what works for us, and for some of the organisations we work with, might help others start these reviews in house.

I hope that we inspired you to do just that :)))

But let's go back to measuring…

5. The Two Worries Worth Answering

The thread raised two worries that stop teams from checking. Both are fair. Neither is a reason to skip it.

"Nobody wants to be the reviewer." Hung is right that checking a colleague's interview can feel political. That's why I'd make the review about the questions and the scoring guide, not about the person. If three interviewers give the same answer a 2, a 3 and a 5, the first thing to fix is the guide. Most of the time, the wording was unclear, not the interviewer. Hung also asked whether this could be a specialist role, someone responsible for "hiring quality". In a bigger team, yes. In a small one, it can be one hour a week for whoever owns the hiring process.

"Won't it feel robotic?" Hung asked Mayur this too: if you count every question, does the interview stop being a real conversation?

I don't think so. I am living proof that it does not feel robotic. My interviews are human to human. I interview like a colleague, sometimes an overly eager one.

Structure decides what you ask and how you score it.

It doesn't stop you from listening, cracking a joke, following up or making the candidate feel at ease.

To me, however, it means every candidate gets the same chance to shine.

6. How to Prove Your Interviews Are Structured

1. Count the questions. Take last month's interviews for one role. Were all the agreed questions asked? Mayur is right: you don't need AI for this. A transcript and 20 minutes will do.

2. Score the same answer three times. Once a week, three interviewers score one recorded answer on their own. Then compare. Big gaps tell you which criteria need clearer wording.

3. Train what each score means. A 1-to-5 scale only works if everyone agrees what a 3 looks like. Write it down, with an example answer for each score.

4. Ask the panel before the debrief. Use Alison's test. If the panel can't name the same three things they were assessing, fix that before the next candidate.

5. Give someone the job. Name the person who reviews the data, and give them the time to do it. Otherwise, as Christopher said, the tool is just an archive.

6. Check the scores against the hires. Six months after a hire, look back at their interview scores. Did the people you scored highest do best? That's the only proof that the structure works, not just that you followed it.

7. Your Checklist

  • We can say which questions every candidate for this role was asked
  • Someone checked last month's interviews against the agreed questions
  • Our scoring guide says what each score looks like, with examples
  • Interviewers are trained on the scale before they interview
  • We compare scores on the same answer at least once a month
  • One named person owns the review, with time to do it
  • We compare interview scores with how hires do after six months

TL;DR: Recording your interviews isn't the same as structuring them. Count the questions, compare the scores and name someone to check. That's your proof.

What would you show if someone asked you to prove your interviews were structured last month?

Want Help Setting Up Scoring and Calibration?

If you'd like help setting up the scoring guide and calibration, our Tech Interviewer Training covers exactly that. And if you'd rather have trained interviewers run the structure for you, that's what our Tech Assessments are for.

Is Interview as a Service Right for You?

Not sure where to start? Our free Buyer's Guide to Interview as a Service explains when handing your technical interviews to a partner makes sense, and when it doesn't.

Sources

This post first appeared in Shortlisted by Kat. Photo by Siora Photography on Unsplash.