I’ve Been Asking the Wrong Interview Question for months (And Here’s What I Finally Learned)

Sometimes the best learning moments come from the most embarrassing realizations.

— -

I’ve been conducting technical interviews for years, and I had this one question I was particularly proud of. I’d lean back in my chair with that slight interviewer confidence and ask: ”Can you walk me through one forward pass in BERT for next token generation?”

Most candidates would pause, maybe look a bit confused, then either stumble through an explanation or dive straight into describing transformer architectures. Some would start explaining attention mechanisms. Others would talk about embeddings and feed-forward layers. A few would admit they weren’t sure.

But here’s the embarrassing truth: not a single candidate ever asked me the follow-up question they should have asked.

It wasn’t until I was discussing this very question with an AI assistant recently that someone finally said: ”Just to clarify — BERT doesn’t actually do next token generation the way you’re thinking.”

And that’s when it hit me like a ton of technical debt.

## The Uncomfortable Realization

For months I’d been asking candidates to explain something that doesn’t actually happen. BERT (Bidirectional Encoder Representations from Transformers) isn’t designed for next token generation at all. It’s a bidirectional model trained for masked language modeling and next sentence prediction, not the autoregressive generation I was thinking of.

I had confused BERT with GPT-style models.

Every single candidate who seemed confused? They were probably right to be confused. Every candidate who stumbled? They might have been trying to politely work around my flawed premise. And every candidate who dove straight into explaining transformer architecture? They were probably doing their best to give me something useful despite the mixed-up question.

## What I Actually Meant to Ask

What I was really trying to test was whether candidates understood autoregressive language generation — the kind that happens in GPT models, not BERT. Here’s the distinction I completely missed:

BERT (Bidirectional):

• Trained with masked language modeling (predicting missing words using context from both directions)

• Can’t generate text sequentially because it sees the entire sequence at once

• Designed for understanding tasks like classification, question answering, sentiment analysis

GPT-style Models (Autoregressive):

• Trained to predict the next token given all previous tokens

• Generate text one token at a time, left to right

• Each new token is fed back into the model to generate the next one

## The Answer I Was Actually Looking For

So here’s what I should have been asking about — a forward pass in an autoregressive transformer for next token generation:

### Step 1: Input Processing

Input: “The weather today is”

Tokens: [101, 1996, 4633, 2651, 2003] # tokenized IDs

### Step 2: Embeddings

• Convert token IDs to dense vectors (token embeddings)

• Add positional encodings to tell the model where each token sits in the sequence

• Result: A matrix where each row represents a token’s embedded representation

### Step 3: Transformer Layers

For each transformer layer:

Self-Attention:

• Each token “looks at” all previous tokens (and itself)

• Computes attention weights: how much should this token pay attention to each previous token?

• Creates a weighted combination of all previous token representations

Feed-Forward Network:

• Passes the attention output through a two-layer neural network

• Applies non-linear transformations to each token representation

### Step 4: Final Prediction

• Take the representation of the last token (“is”)

• Pass it through a final linear layer that outputs scores for every word in the vocabulary

• Apply softmax to get probabilities: maybe “sunny” (0.23), “cloudy” (0.18), “rainy” (0.15), etc.

• Sample or pick the highest probability token as your next word

### Step 5: Autoregressive Generation

• Add the predicted token to your sequence: “The weather today is sunny”

• Feed this new sequence back into the model to predict the next token

• Repeat until you have a complete sentence or hit a stopping condition

## The Deeper Lesson

This mistake taught me something important about technical interviews — and about staying humble in our field. Here’s what I learned:

For Interviewers: Ask questions you can answer confidently yourself. If you’re not 100% sure about the technical details, you’re not just wasting the candidate’s time — you might be unfairly penalizing them for your own confusion.

For Candidates: It’s okay to ask clarifying questions. In fact, it’s often the right thing to do. If something doesn’t make sense, speak up. You might be saving everyone from a confused conversation.

For All of Us: Technology moves fast, and it’s easy to mix up concepts, especially when they’re related. BERT and GPT are both transformers, both deal with language, both use attention mechanisms. The devil is in the architectural details.

## What I’m Doing Differently Now

I’ve updated my interview questions to be more precise:

• Instead of asking about “BERT for next token generation,” I now ask candidates to explain the difference between bidirectional and autoregressive language models

• I ask them to walk through how GPT generates text, step by step

• I make sure I can confidently explain every aspect of what I’m asking before I ask it

And honestly? The conversations are much better now. Candidates can engage more deeply when the question actually makes sense.

## The Silver Lining

This embarrassing realization led to one of my best learning experiences in months. I ended up diving deep into transformer architectures, really understanding the differences between various model types, and becoming a better interviewer in the process.

Plus, it reminded me why I love this field: there’s always more to learn, and sometimes your biggest mistakes lead to your best insights.

— -

What about you? What technical concepts have you realized you didn’t understand as well as you thought? I’d love to hear about your own learning moments in the comments.

And if you’re ever in one of my interviews and I ask a question that doesn’t make sense, please — ask me to clarify. You’ll be doing us both a favor.