Most bias in tech interviews comes from having no plan.

When nobody wrote the questions down, every candidate gets a different interview. When every candidate gets a different interview, you end up comparing conversations. To be more precise, you compare what your interviewers remember of those conversations.

What the Research Says

Structure is what makes an interview work at all.

Structured interviews predict job performance best. In 2022, Paul Sackett and his co-authors re-ran the big numbers on what predicts how well people do in a job. Structured interviews came first and cognitive ability tests came fifth. The same paper found a very wide spread for structured interviews. A process with fixed questions and a scoring guide earns the top score. A manager with four bullet points in a notebook doesn't.

Structure shrinks group differences. A meta-analysis by Allen Huffcutt and Philip Roth looked at how candidates of different races were rated in interviews. The differences were small overall, and smaller still when the interview was structured.

Being watched can cut performance in half. Researchers at NC State and Microsoft ran a randomised trial with 48 computer science students. Some solved a whiteboard problem alone, and some solved it while an interviewer watched. Being watched cut their performance by more than half.

I can relate to that. I was brilliant at maths at school and later in high school. My maths teacher picked me as one of 8 students to compete against other schools in a national competition, because I did so well in class.

What do you think happened?

I scored 3 out of 20. Talk about a maths genius, huh? I was removed from the team, where I was the only girl, and I learned my lesson. I perform very poorly in an unpredictable setting while others sit, watch and judge me.

Back to the study. Every woman who solved the problem in private passed. None of the women who solved it in public did. The sample was small, and the authors say so. But it shows how much the setting alone can decide the result.

So a fair interview needs two things:

  • the same yardstick for everyone
  • a setting where people can actually show what they know.

My 10 Rules

These are the rules behind every technical assessment we run at AssessDev. I've used them in more than 2,000 technical interviews.

1. Define the job before you write a single question

Every role starts with a 30-minute kick-off call. What will this person actually do every day? What's a must-have and what's a nice-to-have? What should they achieve in their first 90 days with you, and in their first 180?

Sometimes the answer is that the company needs a data scientist rather than a CTO. That's based on a true story. If the job is fuzzy, the interview will be too.

2. Write the questions before you meet anyone

We prepare 10 to 12 questions per role. Once the first candidate has been interviewed, the questions don't change. If you invent questions in the room, every candidate sits a different exam.

3. Write down what a good answer sounds like, in advance

We grade every answer as Fail, OK, Great or Exceptional, and we agree what each level means before the first interview. Say I ask how to permanently delete a user and their data. "OK" handles the database records. "Great" also thinks of backups and third-party services. "Exceptional" brings in legal and GDPR. Now two interviewers can grade the same answer the same way.

4. Grade against the role you're hiring for

A mid-level backend developer doesn't need "Exceptional" on architecture. A CTO does. Decide the level you need before you start. Otherwise you'll reject good people for not being someone else, such as the best candidate you've ever met.

5. Give everyone the same agenda, and tell them upfront

Every assessment has the same three parts: an intro, about 30 minutes of questions and a live challenge of about 30 minutes. We send every candidate a preparation email 24 hours before the interview, and we repeat the agenda at the start. There are no surprises and no hidden rounds.

6. Help them relax

We start with a short ice-breaker, so it feels like two colleagues working on a problem.

Our interviewers can help a lot, and we tell candidates that. We explain what we can do for them during the interview, and that asking for help won't count against them. In the live challenge, candidates can ask questions and look things up, as they would at work. The interviewer is there to support them.

After the NC State study, I see this as a rule about accuracy.

7. Give everyone the same AI rule, out loud

Our live challenge is solved without AI tools, and every candidate is told that before it starts. Whatever your rule is, it's only fair if everyone hears the same one. I wrote more about this in Post Your AI Interview Rules.

8. Take notes during the interview, every time

Memory is not evidence. We write down what the candidate said and did while it happens, and clean up the notes right after. Notes let you explain a decision later, to the hiring manager or to the candidate.

9. Score each answer on its own, before you talk to anyone

Score question by question, before the debrief. Otherwise the loudest person in the room decides. Gut feeling is allowed, but it comes last and counts as one input.

That's the Kahneman method. Score the parts first, then add your overall impression as one more score. For those of you who know me, I'm a big Kahneman fan.

10. Give every candidate feedback

Our report is ready within 24 hours, with feedback the client can send as it is. During the interview, when someone gets an answer wrong or only half right, I tell them the right answer. It takes a minute, and the interview becomes useful for them even if they don't get the job.

What Candidates Notice

Rule 10 is the one people find most surprising. So here's what it looks like from the other side.

A candidate once ended our interview by saying, "I enjoyed talking with you, and thank you for your suggestions!" The "suggestions" were the full answers to the questions she hadn't fully answered. She didn't call it an interrogation. She called it a conversation.

A data engineer, David O., wrote this about his interview with us: "I felt I could relax and focus on the task, without worrying about my fears which normally make me perform badly. That allowed me to think, read carefully and show my real skills."

That last part is the whole point. A fair interview is one where you see the candidate's real skills. The nerves, the small talk and the interviewer's mood that day get out of the way.

Don't take my word for it. Read what candidates have written about us.

The Honest Trade-Off

Ten rules take work up front. The kick-off call, the questions and the grading guides take a few hours per role before the first candidate walks in. Fixed questions also mean you sometimes can't chase an interesting answer as far as you'd like. We leave room for follow-up questions, but every candidate gets the same core questions.

In our experience, those few hours come back quickly. Debriefs get shorter, decisions get easier to defend, and fewer good candidates slip through.

Where to Start This Week

If ten rules feel like a lot, start with three:

  1. Fix the questions (rule 2). Write 10 to 12 questions for one open role and use them for every candidate.
  2. Write the grading guide (rule 3). For each question, write one line on what OK, Great and Exceptional sound like.
  3. Score before the debrief (rule 9). Ask every interviewer to submit their scores before anyone talks.

This is an afternoon of work. Possibly less.

A Quick Checklist

  • The role is defined, with must-haves and nice-to-haves, before any questions are written.
  • 10 to 12 questions are written and fixed before the first candidate.
  • Every question has a written grading guide (Fail, OK, Great or Exceptional).
  • We know which level the role actually needs.
  • Every candidate gets the same agenda, in advance and at the start.
  • The interview starts with an ice-breaker, and the challenge allows questions.
  • Every candidate hears the same AI rule before the challenge.
  • The interviewer takes notes during the interview.
  • Scores are written per question, before the debrief.
  • Every candidate gets written feedback.

So, honestly, which of these 10 rules does your team skip most often, and why?

Sources

This post first appeared in Shortlisted by Kat. Photo by Kelly Sikkema on Unsplash.

Interview Like This In-House

Want your own engineers to interview like this?

Tech Interviewer Training

Two mornings of live training on structure, question banks, feedback and live challenges for your engineers and hiring managers.

Sample Interview Structure

Start for free with the exact interview structure we use in every tech assessment.