When the Evaluator Becomes the Attack Surface.

An adversarial perspective on evaluating AI systems that evaluate…

Most of the AI security work I’ve come across focuses on the model itself.

Can I jailbreak it? Can I inject instructions into it? Can I make it leak information? Can I make it call a tool it shouldn’t?

Those are all important problems. But recently I’ve been thinking about another part of the system that doesn’t get as much attention:

What if the thing judging the AI can also be manipulated?

That’s where this idea started for me.

The evaluator is part of the system

Imagine a fairly simple setup:

User → AI → Answer → Evaluator → Decision

The evaluator could be another LLM, a critic, a verifier, a reward model, or even a rules-based system. Its job is basically to answer questions like:

“Is this answer correct?” “Is this response safe?” “Should we accept this output?”

We normally treat that evaluator as a trusted component. But from a security perspective, that doesn’t really make sense. It’s still software, It still has inputs. And if an attacker can influence those inputs, then the evaluator should probably have a threat model too.

The interesting attack isn’t necessarily making the AI produce something obviously malicious.

It could be making the evaluator believe that a bad answer is a good one.

Correctness vs evaluated correctness

This distinction is what I find most interesting.

Suppose an AI produces an incorrect answer.

If the evaluator correctly identifies it as incorrect, the system has behaved as expected. But what happens when the evaluator gives that same answer a high score?

Now we have two different things:

Actual correctness

and

what the system thinks is correct.

If those start diverging in a predictable way, there’s potentially a security problem.

An attacker doesn’t necessarily need to break the underlying model. They may only need to influence the mechanism deciding whether the model’s output should be trusted.

You don’t necessarily need a jailbreak

When I think about this, I don’t think the attack has to look like a traditional prompt injection.

There are plenty of other things an attacker might be able to influence:

• The evidence provided to the evaluator
• The order in which evidence appears
• Retrieved documents
• Source selection
• The candidate answer itself
• Confidence cues
• Formatting
• Contradictory information
• Instructions hidden inside untrusted content

For example, imagine an AI is asked whether a particular claim is supported by evidence. The correct answer is:

“There isn’t enough evidence.”

Now imagine an attacker can introduce several sources repeating the same unsupported claim.

The evaluator might see:

Source A says X. Source B says X. Source C says X.

It may interpret this as strong evidence.

But what if all three sources ultimately came from the same incorrect claim?

Three copies of the same information don’t necessarily mean three independent pieces of evidence.

That creates an interesting attack surface.

Being confidently wrong is worse

This also made me think about how we evaluate these systems.

Accuracy is obviously important. But I don’t think accuracy alone tells the whole story. Imagine two evaluators that are both 90% accurate. One makes mistakes fairly randomly.

The other becomes extremely confident whenever it encounters a particular adversarial pattern.

From an attacker’s perspective, the second system is much more interesting.

Random failures are difficult to exploit. Systematic failures aren’t.

If I can discover a property that reliably makes an evaluator approve an incorrect answer, I’ve got something much more useful than a one-off model mistake.

What about confidence?

This is where calibration becomes important.

If a system says it is 80% confident, we’d ideally expect it to be correct around 80% of the time for similar predictions.

But what happens when we deliberately change the input to make it adversarial?

Does the confidence remain meaningful?

Or does the evaluator become highly confident for reasons that have nothing to do with whether the answer is actually correct?

That’s something I’d want to measure rather than simply assuming the evaluator is reliable.

Sometimes the right answer is “I don’t know”

There’s another behavior I think is particularly important in security-sensitive AI systems:

abstention.

Imagine two systems. The first answers every question. The second answers when the evidence is sufficient and says:

“Insufficient evidence.”

when it isn’t.

The second system might look worse if we’re only measuring how many questions it answers.

But in a security-sensitive environment, I’d much rather have a system admit that it doesn’t have enough evidence than confidently approve something unsupported.

The goal shouldn’t be to make AI refuse everything.

It should be able to recognize when it shouldn’t be confident.

So how would I actually test this?

I’d start fairly simply.

First, create a set of questions where we know the ground truth. Then establish how the evaluator behaves under normal conditions. After that, create adversarial versions of the same scenarios.

For example:

• Add misleading evidence
• Remove important evidence
• Introduce contradictory evidence
• Add irrelevant but authoritative-looking information
• Repeat the same claim through multiple sources
• Change the ordering of evidence
• Introduce instructions into untrusted content

The important part is that the underlying truth doesn’t change.

Then we can ask:

Did the evaluator change its decision because the evidence actually became stronger, or because we changed how the evidence was presented?

That’s the experiment I’d be interested in.

One failure isn’t enough

I also wouldn’t call something a vulnerability just because I managed to fool a model once. LLMs make mistakes. That’s expected.

I’d want to see whether the behavior is repeatable.

I’d vary:

• The wording
• The prompt
• The evidence
• The task
• The model
• The evaluator

If the same adversarial property continues to produce the same type of failure, that’s much more interesting.

At that point we’re moving from:

“The model got this example wrong.”

to:

“We’ve found a systematic way of influencing the evaluation.”

That’s a much stronger security finding.

What happens after you find it?

Finding an attack is only half the job. The mitigation depends on what caused the failure.

If untrusted instructions are getting mixed with trusted evaluation instructions, we need better trust boundaries. If the problem is source manipulation, we may need stronger provenance and source-quality controls.

If repeated sources are being treated as independent evidence, we need to account for source correlation. If the evaluator is simply overconfident, calibration and abstention mechanisms may help.

And whatever we discover should become a regression test.

Otherwise there’s a good chance the same weakness comes back later.

What about using multiple evaluators?

A natural response is:

“Why not just use three evaluators instead of one?”

That can certainly help.

But three evaluators aren’t automatically three times as secure. If all three share the same weakness, an attacker may still be able to exploit it.

I’d be more interested in understanding whether their failure modes are independent.

If they all fail for the same reason, majority voting doesn’t really solve the underlying problem.

Is this just prompt injection?

I don’t think so.

Prompt injection can definitely be part of the attack. But I’m thinking about something slightly broader:

Evaluation manipulation.

The attacker isn’t necessarily trying to control what the model says.

They’re trying to influence how the system judges what the model says.

That could involve prompts, retrieved information, evidence, source relationships, formatting, or biases in the evaluator itself. The common objective is the same:

Make the evaluation diverge from reality.

Where I think this gets interesting

As AI systems become more complicated, we’re going to see more chains like:

Model → Retrieval → Tools → Agent → Critic → Verifier → Decision

Every one of those components introduces assumptions.

As security engineers, we’re used to asking:

• What can the attacker control?
• What’s trusted?
• Where are the trust boundaries?
• What happens if one component behaves unexpectedly?
• Can we reproduce the failure?
• Can we build a regression test?

I think we need to ask the same questions about AI evaluation systems.

Because the attack surface isn’t always the model.

Sometimes, the thing telling us whether the model is safe can itself be the thing we need to attack.

And that’s probably the part of AI security I find most interesting right now..