How I Built a Domain-Specific LLM Evaluation Benchmark for a UE5 Industrial Simulation Agent

While working on an industrial simulation project using Unreal Engine 5 and an AI Agent, I encountered a practical challenge:

How can I construct an LLM evaluation benchmark tailored to this project domain, and how can I decide whether SFT is actually necessary?

This question became more important than simply asking whether the LLM “worked.”

In a simulation control system, an LLM does not just generate text. It selects tools, fills arguments, handles ambiguous requests, rejects invalid commands, and interacts with a running simulation environment.

A generic benchmark cannot answer whether the agent is reliable for this specific project.

So I built a domain-specific evaluation benchmark for the project and used it to guide the engineering decisions around validation logic, serving backend, prompt design, and SFT.

Project Context

The project, V-CORE, is an on-premise LLM Agent system connected to a UE5-based industrial simulation.

The agent receives natural language commands and routes them into simulation actions such as:

• moving AGVs
• running station tasks
• changing simulation speed
• querying simulation state
• evaluating KPI acceptance conditions
• generating reports

At first, the system appeared to work in simple demos.

However, when I started testing more realistic commands, several failure cases appeared:

• invalid JSON output
• wrong tool selection
• missing arguments
• incorrect argument values
• ambiguous commands executed without clarification
• negative-control prompts incorrectly triggering tools
• KPI conditions parsed incorrectly

At this point, it was tempting to conclude:

The model is weak. I should fine-tune it.

But that would have been a guess.

Before deciding on SFT, I needed a benchmark that could tell me where the actual problem was.

Why a Generic LLM Benchmark Was Not Enough

General LLM benchmarks measure broad capabilities such as reasoning, coding, knowledge, or instruction following.

But my project needed answers to more specific questions:

1. Can the model choose the correct simulation tool?
2. Can it fill the correct arguments?
3. Can it reject invalid requests?
4. Can it ask for clarification when the command is ambiguous?
5. Can it handle Korean and English operator commands?
6. Does the validation layer actually improve reliability?
7. Is fine-tuning necessary, or are prompt and system fixes enough?

For this project, “the model answered well” was not a useful metric.

The benchmark had to measure whether the agent produced the right executable decision.

Phase 1: The First Evaluation Was Too Small

My first evaluation was a small smoke test with 12 prompts.

It was useful for checking whether the evaluation harness worked, but it was not reliable enough for making engineering decisions.

There were several problems:

• 12 samples were too few.
• One prompt changed the score by 8.33 percentage points.
• Each category had too few cases.
• There were no repeated runs.
• The test checked tool selection, but not argument correctness.
• There was no confidence interval.

This meant I could not responsibly say:

Provider A is better than Provider B.

or:

The model needs fine-tuning.

The evaluation itself had to be improved first.

Phase 2: Building a Domain-Specific Benchmark

I redesigned the benchmark around the actual tasks of the simulation agent.

The new benchmark had:

• 133 labeled cases
• 12 task categories
• Korean and English prompts
• 5 repeated runs per case
• argument-level scoring
• static JSONL test cases
• Wilson 95% confidence intervals

The benchmark categories included:

• positive tool invocation
• negative control
• ambiguous command
• single-parameter extraction
• multi-parameter extraction
• missing parameter handling
• invalid parameter handling
• KPI acceptance parsing
• disambiguation between similar tools
• long natural language commands
• sequential workflow
• state-dependent commands

The most important change was argument-level scoring.

Instead of only checking whether the model selected the right tool, the benchmark checked:

tool_correct
args_correct
json_ok
schema_ok
task_success

The headline metric was:

Task Success = correct tool + correct arguments + valid schema or acceptable fallback

This better matched the real product question:

Did the agent make the right executable decision?

Separating Model Performance from System Performance

The production system was not just a raw LLM call.

It included a validation layer:

model output
→ JSON extraction
→ schema validation
→ repair retry
→ rule-based fallback
→ final action or decline

At first, I assumed this layer would improve reliability.

But instead of assuming, I tested it.

I designed a 2×2 ablation:

ConditionMeaningA1Ollama onlyA2Ollama + Validation LayerB1llama.cpp onlyB2llama.cpp + Validation Layer

This produced:

133 cases × 5 repeats × 4 conditions = 2,660 scored runs

This structure separated two questions:

1. How capable is the model/provider by itself?
2. How much does the validation layer help or hurt?

This was one of the most valuable parts of the benchmark.

It showed that system-level reliability cannot be inferred from model-level behavior alone.

Finding a Validation Layer Bug Through Evaluation

The benchmark revealed that the validation layer was not automatically helpful.

In one path, the production system became worse after adding validation.

The issue was the repair-retry prompt.

The retry prompt was too aggressive. It pushed the model to always produce a valid tool call, even when the correct behavior was to decline or ask for clarification.

This caused failures in cases such as:

• negative-control prompts
• ambiguous commands
• missing-parameter commands

The fix was to make decline a valid terminal state.

After that, the validation layer no longer forced tool calls when the safe behavior was to stop.

I also added stricter range validation for tool arguments, so invalid values such as negative station IDs or invalid speed values could not pass through.

After the fix, the validation path improved significantly:

54.3% → 75.9% task success

This was the first major lesson:

A validation layer is not automatically a reliability layer. It is software, and it must be evaluated.

Using the Benchmark to Decide Whether SFT Was Necessary

Once the validation layer was fixed, the next question was:

Do I still need SFT?

The benchmark helped separate the remaining problems.

Some failures looked like model limitations at first, but turned out to be cheaper system-level problems.

For KPI acceptance parsing, the issue was not that the model lacked the ability. The prompt simply did not specify the expected acceptance structure clearly enough.

Adding a small amount of explicit instruction dramatically improved the KPI category.

For disambiguation, prompt changes did not help much. However, changing the serving configuration did.

Using llama.cpp with reasoning disabled improved both latency and disambiguation behavior under the same model weights.

This changed the SFT decision.

SFT was not necessary for basic accuracy at that stage.

The benchmark showed that the higher-ROI fixes were:

• prompt correction
• validation-layer correction
• serving configuration
• better schema handling

This was the second major lesson:

Fine-tuning should be justified by evaluation, not by instinct.

Reframing SFT: From Accuracy Improvement to Prompt Distillation

Even though SFT was not necessary for basic accuracy, the system still had another problem.

The routing behavior depended heavily on a long production prompt.

That created maintenance debt:

• Every tool change required prompt editing.
• Prompt length increased as tools were added.
• Routing behavior lived in text, not weights.
• The model became dependent on a fragile instruction stack.

So I reframed the purpose of SFT.

The goal was not:

Improve raw accuracy because the model is weak.

The goal became:

Distill the routing behavior from a long prompt into the model weights.

I trained a small QLoRA adapter for tool routing and evaluated it using a three-condition setup:

ConditionWeightsPromptTool-Routing SuccessBase + Minimalbase model4-line prompt12%Base + Fullbase modellong production prompt49%SFT + Minimalfine-tuned model4-line prompt96%

This result showed that SFT successfully moved routing behavior from the prompt into the adapter.

The important result was not just the 96% score.

The important result was that the model no longer depended on the long prompt.

This was the third major lesson:

SFT can be valuable even when it is not needed for initial accuracy. It can reduce prompt dependency and maintenance cost.

What This Benchmark Changed

The benchmark changed the way I made decisions in the project.

Without the benchmark, I might have made simplistic conclusions:

• “The model is bad.”
• “The provider is bad.”
• “The validation layer helps.”
• “We need fine-tuning.”

With the benchmark, each decision became more precise:

• The validation layer had a retry-policy bug.
• Some KPI failures came from prompt underspecification.
• Some disambiguation failures came from serving behavior.
• SFT was not needed for basic accuracy.
• SFT was useful for prompt distillation.

This made the project more engineering-driven and less intuition-driven.

Key Takeaways

This project taught me that LLM evaluation for agents must be domain-specific.

For a UE5 industrial simulation agent, I needed to evaluate not only language quality, but executable decision quality.

The most important takeaways were:

1. Generic benchmarks are not enough for domain agents.
2. Tool selection alone is not a sufficient metric. Arguments must be scored.
3. Validation layers can improve or damage reliability.
4. Model performance and system performance must be separated.
5. Fine-tuning should be a measured decision, not a default reaction.
6. SFT can be reframed as prompt distillation, not just accuracy improvement.

The final value of the benchmark was not just a score.

It became a decision-making tool.

It helped answer:

What should I fix first: the prompt, the schema, the validator, the serving backend, or the model weights?

That question is more useful than simply asking whether the LLM is good or bad.