Mercor AI Interview Part 3 : AI Engineer Role

I was free on a Saturday, the kind of free where you’ve already doom-scrolled through every social media app twice, and you’re just sitting there questioning your life choices. So naturally, I logged into Mercor and signed up for another interview.

If you’ve read my previous two posts (Link for part 2 → Here!), you know the drill — Mercor uses AI interviewers. No humans on the other side. Just you, a browser tab, and an AI that somehow asks better questions than most humans I’ve met in real interviews.

This time, the role was AI Engineer.

Now, important context: I’m not an AI engineer. I’m a backend developer who has read enough Hacker News to sound like one at parties. But curiosity got the better of me. How hard could it be?

(Narrator: It was, in fact, quite hard.)

you asked me what?

The Interview Structure

Same format as before — 30 minutes, divided into sections:

• Resume Discussion (5–7 minutes) — The interviewer asked about my past projects. Specifically, it latched onto the fact that I’d worked with distributed systems and Kafka-like pipelines, and asked how I’d think about reliability in ML pipelines. I improvised. Confidently.
• System Design (15 minutes) — The main event. More on this below.
• Feature Expansion (5 minutes) — “Now make it smarter.”
• Behavioral (5 minutes) — The usual “how do you learn things” question that I genuinely enjoy answering.

The System Design Problem

Design a real-time AI code review system.

The scenario: you’re building a backend for a product that automatically reviews pull requests using an LLM. Developers open a PR, and within a minute, they get AI-generated feedback — inline comments, a summary, and a severity score.

The constraints:

What it needs to do:

• Accept PR webhook events from GitHub/GitLab
• Fetch the diff, send it to an LLM for review
• Post comments back to the PR via the Git provider’s API
• Store review history per repo, per user
• Support multiple LLM providers (OpenAI, Anthropic, local models)

The hard parts:

• LLM calls are slow (5–30 seconds per review) — can’t block the webhook response
• Diffs can be huge — 10,000+ line PRs exist (unfortunately)
• Cost control — you can’t send a 10k line diff to GPT-4 and pretend money isn’t real
• Multi-tenancy — thousands of repos from different orgs, all hitting the system simultaneously

My Approach

I broke it down component by component, which is the only sane way to approach system design without spiraling.

1. Webhook Ingestion Layer

When GitHub fires a webhook for a new PR, you need to respond within a few seconds or GitHub marks it as failed and retries. So the ingestion layer does exactly one thing: accept the event, validate the signature, and drop it into a queue. That’s it. No LLM calls here, no DB writes, nothing fancy.

I proposed using a simple HTTP server fronted by a job queue — something like a Redis-backed queue or a lightweight message broker.

2. Diff Processing & Chunking

This is where it gets interesting.

You can’t just take a 500-file diff and shove it into an LLM prompt. Context windows have limits, and even when they don’t, you’re paying per token. I proposed a chunking strategy:

• Split the diff by file
• For each file, further split by logical hunks (the @@ sections in a git diff)
• Prioritize chunks by change size and file type — a 3-line change to auth.go is probably more important than a 200-line auto-generated protobuf file
• Run chunks in parallel across multiple LLM calls
• Aggregate results into a unified review

The interviewer asked: “How do you ensure the aggregated review is coherent and not just five disconnected comments?”

Good question. I said you’d run a final “summarization pass” — take all the chunk-level feedback, send it back to the LLM with a prompt like “synthesize these into a coherent review summary.” One extra LLM call, but it ties everything together.

3. LLM Abstraction Layer

Supporting multiple providers (OpenAI, Anthropic, local Ollama instances) means you need a clean abstraction. I proposed a provider interface:

type LLMProvider interface {
    Complete(ctx context.Context, prompt string, opts Options) (string, error)
}

Each provider implements this. The caller doesn’t care whether it’s GPT-4 or Claude or a fine-tuned Llama running on someone’s gaming PC. You can also add fallback logic here — if OpenAI is rate-limiting you, fall back to Anthropic automatically.

The interviewer liked this and asked about cost tracking. I said each provider implementation wraps the call with token counting, and you write usage logs to a DB. Orgs have monthly budgets; when they hit the limit, you downgrade them to a cheaper model or queue their reviews.

4. Result Delivery

Once the review is ready, you need to post it back to GitHub as PR comments. This is the most failure-prone part — GitHub’s API can be flaky, rate limits exist, and you don’t want to lose a review because of a transient 503.

I proposed async delivery with retries:

• Store the generated review in DB first
• A separate worker picks it up and attempts delivery
• Exponential backoff on failures
• If delivery fails 5 times, mark it as failed and notify the user via email instead

The key insight I mentioned (which I learned from the previous Mercor interview, honestly): separate the critical path from the delivery path. Generating the review is critical. Posting the comment is best-effort with retries.

The Latency Challenge

“You promised reviews within a minute. How do you actually guarantee that?”

This one required thinking through every step:

• Webhook to queue: < 1 second (just an HTTP accept + enqueue)
• Queue to worker pickup: < 2 seconds (worker polling or push-based queue)
• Diff fetch from GitHub API: 2–5 seconds
• LLM calls (parallelized across chunks): 10–25 seconds
• Aggregation pass: 5–8 seconds
• Delivery to GitHub: 2–5 seconds

Total: roughly 20–40 seconds in the happy path. Under a minute, but not by a lot.

For large PRs, I said you’d progressively deliver — post chunk-level comments as they’re ready, then add the summary at the end. Users see feedback appearing in near real-time rather than waiting for the whole thing.

The interviewer nodded (metaphorically — it’s an AI, but it said something like “that’s a reasonable tradeoff”). Progress.

Feature Expansion

“Users love it. What do you build next?”

I suggested two things:

1. Review Memory — Store past reviews per repo. When the same kind of bug appears again (say, someone keeps forgetting to handle nil errors in Go), the system references past feedback: “Hey, we flagged this pattern in PR #47 too.”Makes the review feel less generic.

2. Severity-Based Routing — Not every PR needs GPT-4. A one-line config change doesn’t need the expensive model. Route by estimated complexity: small diffs go to a fast, cheap model; large architectural changes go to the best available model. Saves money, maintains quality where it matters.

The interviewer asked how I’d estimate complexity before running the review. I said: file count, total lines changed, presence of test files (more tests = probably more complex logic), and cyclomatic complexity if you can run a static analysis pass cheaply upfront.

It seemed satisfied. I was relieved.

The Behavioral Round

Same question as last time, more or less — how do I learn new things?

I gave basically the same answer (read docs, build something small immediately, read others’ code, write about it). But I added something new: I learn a lot from interviews themselves. The Kafka answer I gave in my Part 2 interview? I read a thread about Kafka literally 10 minutes before that interview. This interview? I’d been reading about RAG systems the night before, which helped me think about chunking.

Preparation doesn’t always have to be months in advance. Sometimes it’s 10 minutes and a Twitter thread.

(I did not mention that part to the interviewer.)

What I Took Away

The AI Engineer interview was harder than the backend one, not because the system design was more complex, but because the problem space is less familiar. With stock price alerts, I could reason from first principles — queues, in-memory matching, WebSockets. With LLM pipelines, there are new failure modes (model timeouts, context limits, token costs) that you don’t encounter in traditional systems.

But the thinking process is the same: break it into components, reason about failure at each step, separate fast paths from slow ones, and make explicit tradeoffs instead of hand-waving.

If you’re preparing for AI engineer roles, I’d suggest getting comfortable with:

• How LLM APIs work (context windows, token limits, streaming)
• Chunking and retrieval strategies (even a surface-level understanding helps)
• Cost as a first-class constraint (not an afterthought)
• Async job processing — because LLM calls will always be too slow for synchronous flows

Still waiting to hear back. At this point, giving Mercor interviews has become my Saturday hobby.

Will update if something actually happens. Until then — back to the backend.