I Got Tired of “It Makes Your Agent Better.” So I Measured It.

How I built a reproducible benchmark for AI tooling and what it revealed about the boring production concerns my data agent kept skipping.

The itch

I have a reflex, probably shared by most data engineers, of not believing numbers I can’t reconcile. It’s the core of the job: we check row counts against the source, we test for duplicate keys, we prove a job is idempotent before we let it run twice. So there’s something that genuinely irritates me about the AI tooling that keeps landing on my desk. Every one of them promises to make my agent “smarter” or “more reliable,” and not one of them hands me a way to verify the claim.

After enough of these, I decided to stop complaining and build the thing I kept wishing existed: a small, reproducible benchmark that measures, concretely, whether a given capability improves an agent’s output. I’ll walk you through how I built it, what it told me, and because a benchmark you can’t criticize is worthless where it can mislead you. The domain is data engineering, because the concerns an agent forgets there (idempotency, backfill safety, reconciliation) are precisely the ones that turn into 2 a.m. incidents.

Step 1: Define “good” before you measure it

You can’t measure quality you haven’t pinned down. So I started by writing, as a flat binary checklist, the concerns a strong engineer would address for a given task. For “load yesterday’s orders into the warehouse”:

ORDER_PIPELINE_CONCERNS = [

“idempotent_writes”, *# rerun doesn’t double-count*

“late_arriving_data”, *# records arriving after the window*

“schema_evolution”, *# tolerates an added column*

“reconciliation_check”, *# validates counts vs source*

“bounded_backfill”, *# backfill targets a specific window*

“publish_pause”, *# downstream paused during backfill*

“rollback_path”, “lineage”, “retention”, “pii_handling”,

]

Repeat across a handful of representative tasks and the total becomes your denominator. When I did this thoroughly, my task set came to 67 distinct concerns.

Step 2: Score responses the same way every time

Then I needed a function to decide which concerns a given agent response actually covered. I deliberately started with deterministic keyword-and-structure detection rather than a fancy judge, because reproducibility matters more than nuance when your goal is a clean before/after delta:

def score(response: str, concerns: list[str]) -> set[str]:

text = response.lower()

signals = {

“idempotent_writes”: [“idempotent”, “merge into”, “upsert”],

“reconciliation_check”: [“reconcile”, “row count”, “control total”],

“bounded_backfill”: [“bounded”, “partition range”, “specific window”],

“publish_pause”: [“pause publish”, “hold downstream”],

“rollback_path”: [“rollback”, “revert”, “restore previous”],

*# one entry per concern…*

}

return {c for c in concerns if any(s in text for s in signals.get(c, []))}

It’s a rough judge. That’s fine it’s the same rough judge applied to both conditions, so the difference between them carries real signal.

Step 3: Run it on and off

Same tasks, run twice once with the capability under test, once without, everything else held constant:

def run(tasks, call_agent, with_skills):

covered = possible = 0

for t in tasks:

prompt = (t[“preamble”] + “\n\n” + t[“prompt”]) if with_skills else t[“prompt”]

covered += len(score(call_agent(prompt), t[“concerns”]))

possible += len(t[“concerns”])

return covered, possible

I pointed this at a real data-engineering skill registry to see whether its “skills” did anything measurable. The output was bracing:

Baseline: 23/67

With skills: 67/67

Delta: +44

Without the skills, the agent addressed about a third of the production concerns. With them, all of them. And here’s the part that reframed the whole thing for me: the skills didn’t make the model smarter. It already knew what idempotency was and why backfills are dangerous. It simply didn’t apply that knowledge until a procedure put it in front of the agent at the right moment. The intervention bought completeness, not intelligence and in data engineering, completeness is most of what separates a demo from a system you trust.

Step 4: Put it in CI

A benchmark you run once is a screenshot. A benchmark in CI is a control. I store the result and fail the build on any regression:

import json, sys

r = json.load(open(“benchmark-results.json”))

if r[“with_skills”] <= r[“baseline”]:

print(“FAIL: change didn’t improve coverage”); sys.exit(1)

print(f”PASS: +{r[‘with_skills’] — r[‘baseline’]}”)

Now if anyone edits a skill or prompt and coverage drops, the build tells me before production does.

Step 5: Attack your own benchmark

This is the step people skip, and it’s the one that keeps you honest. Keyword scoring is gameable an agent that writes “idempotent” without implementing it scores a false positive. For the concerns that genuinely matter, I upgrade to a structural check (does the generated SQL actually contain a MERGE?) or an LLM judge with a strict rubric and a held-out validation set. And I had a colleague review the concern list, because mine inevitably encoded my own blind spots. The number is exactly as trustworthy as the rubric underneath it, no more.

Conclusion

Building this changed how I evaluate AI tooling, full stop. I no longer accept “it makes your agent better” as a claim; I ask for the benchmark, ask to reproduce it, and ask what’s on the concern list and who reviewed it. Tools doing real engineering can answer those questions. Tools selling a feeling go quiet. And the best part is independence once you’ve built your own harness, you can measure the next tool yourself in an afternoon and never outsource the judgment again.

For data teams specifically, there’s an immediate payoff too: the benchmark surfaces exactly which production concerns your agent is dropping right now, which tells you precisely what to fix. Start with the concern checklist; everything else builds on it.

If you’d like to see a fully assembled version rubric, eval runner, and CI gate built around a real skill catalog the open-source data-engineering-agent-skills project ships exactly this benchmark, and it’s where the 23-to-67 result here came from.

If you wrote your own concern checklist, what would lead the list — the thing your agent forgets every single time?