You need a licence to drive a car. You don’t need one to decide who gets a job.

In most companies, all it takes to interview a candidate is a free slot in your calendar. Nobody checks whether you know the questions, the scoring guide or what a good answer sounds like.

OpenAI has decided that isn’t good enough for at least one of its interviews.

1. What’s happening

Fortune’s Emily Forlini reported on 5 October that OpenAI is piloting a mission interview. I wrote about it last week too, if you remember 😉

The interviewer comes from a different team than the one the candidate applies to. And OpenAI is building a training programme that staff complete to become certified to run these interviews.

Unfortunately, that’s about all we know.

Fortune says the programme is “just starting to ramp up”. It doesn’t say what the training covers, how long it takes or how many people are certified so far.

The best documented version of this idea is older.

Amazon’s Bar Raisers are interviewers who go through a certification of their own.

They are nominated by their manager, peers or other Bar Raisers. They get extensive training on Amazon’s 16 Leadership Principles. Then they shadow experienced Bar Raisers for several months.

Amazon says some people graduate after fewer than a dozen interviews, and some need 40 or more 🤯.

So two of the biggest employers in tech treat interviewing as a skill you have to earn.

Not something you are born with.

2. Why most teams skip it

Most interviewers have never been trained.

Only one in three interviewers had formal training. Derek Chapman and David Zweig surveyed 592 interviewers from more than 500 organisations (University of Toronto, 2005; the paper is in Personnel Psychology). The trained third favoured more structure in their interviews. As Zweig put it:

“That’s way too low. Everyone should receive formal training on how to conduct a structured interview.”

The study is 20 years old, so the exact share has surely moved. But I’d be surprised if it moved much in tech.

Here’s how it usually goes.

A good engineer becomes senior. Senior engineers interview. Nobody shows them how. They copy whatever they remember from their own interviews, good or bad, and every candidate gets a slightly different version of it.

Interviewing looks easy because everyone has been interviewed. That’s exactly why nobody trains for it.

It’s like going to the doctor. Everyone can, and must at some point in their lives, see a doctor. A few selected ones become doctors, and that is after many, many years of relentless studying.

You do not need a decade of study and practice to become a good interviewer. You do, however, need training to become one.

3. What I see in my own work

I run technical interviews every week for other companies, so I see the same pattern from the outside.

Some tech interviewers spend days preparing to conduct their interviews. Some are excited to be chosen to do so. Some are unhappy that they’ve been pulled from project work. There is usually no fixed process to help them bridge the gap between a great engineer and a great interviewer.

By the way, what makes an interviewer reliable is boring.

Every candidate in a role gets the same questions. The scoring guide is written before the first interview, with scorecards for each question and task. Notes are taken during the interview, while it happens. Every candidate gets written feedback within 24 hours.

None of that comes naturally, and all of it can be taught.

4. The honest part

A certificate can turn into a slide deck people click through once. That’s worse than nothing, because it makes everyone feel trained.

Amazon’s version takes months. A 40-person startup can’t copy that, and it doesn’t need to.

And training can’t rescue bad questions. If the interview itself has no structure, a certified interviewer just runs an unstructured interview with more confidence.

It’s also worth noticing where OpenAI started.

A “mission” or values round is the easiest one to turn into a vibe check. That’s exactly the round where you want trained interviewers with a scoring guide.

5. How to certify your own interviewers

A light version you can run with a team of any size:

1. Write down what they must know. The skills the role needs, the questions, the scoring guide and your AI rule. If it isn’t written down, there’s nothing to train against.

2. Shadow first. The new interviewer sits in on two or three interviews and scores silently, next to an experienced interviewer.

3. Then swap seats. The new interviewer leads, and the experienced one watches and scores too.

4. Compare scores before they go solo. If their scores are far from the experienced interviewer’s, talk through why. Most gaps come from reading the scoring guide differently.

5. Check their feedback. Read the written notes and the candidate feedback from their first interviews. Vague feedback usually means vague scoring.

6. Check again when the questions change. A new question set means a short new round of calibration.

Prefer to Get Training From an Organisation Which Has Done This Before?

AssessDev’s got you covered.

Stop losing great candidates to bad interviews. Give your hiring managers and interviewers the structure, confidence, and skills to hire right, every time.

Tech Interviewer Training

Two mornings of live training on structure, scoring and feedback for your engineers and hiring managers.

But if you want to prepare and train your interviewers yourself, follow these steps.

6. Your checklist

  • Every interviewer has the questions, the scoring guide and the AI rule in writing
  • New interviewers shadow before they lead
  • They lead at least once with an experienced interviewer watching
  • We compare their scores with an experienced interviewer’s
  • Someone reads their first written feedback
  • We re-check interviewers when the questions change
  • Values or “fit” rounds have a scoring guide and trained interviewers

TL;DR: OpenAI is certifying interviewers, and Amazon has done it for years. You don’t need months of training, but nobody should interview alone before they’ve shadowed, been watched and compared their scores.

Who taught you how to interview, if anyone?

Sources

This post first appeared in Shortlisted by Kat. Photo by Erik Witsoe on Unsplash.