The Agent That Didn’t Know What It Didn’t Know

What building a production data agent taught us about the real gap between AI demos and AI systems

There’s a moment every engineer working with AI eventually hits. You’ve got a demo. It works beautifully. You show it to someone who matters. They ask a slightly different question than the one you prepared for — and the whole thing falls apart.

Not because the model is bad. Because you built a demo, not a system.

This is the story of what it takes to close that gap.

The Task We Chose to Not Hide From

We set out to build a data analytics agent — one that could answer complex business questions against real enterprise databases. Not a toy. Not a carefully prepared single-table demo. The kind of messy, multi-database, semantically ambiguous environment that actual enterprise data teams deal with every day.

We benchmarked against DataAgentBench (DAB), a research benchmark from UC Berkeley’s EPIC Data Lab. It’s the first benchmark designed to test AI data agents against realistic enterprise workloads: 54 queries across 12 datasets spanning 9 domains, running against four different database systems — PostgreSQL, MongoDB, SQLite, and DuckDB — often within the same query.

The best current score on DAB? 54.3% pass@1. That’s the state of the art. More than four in ten queries, even the best systems get wrong.

That number isn’t a flaw in the benchmark. It’s a signal about the gap between what raw language model capability can do, and what it takes to engineer a reliable data agent.

Why Data Agents Are Hard (and It’s Not What You Think)

The naive assumption is that data agents fail because the language model can’t write good SQL. That’s rarely the real problem. Modern LLMs can write reasonable SQL. What they can’t do — without careful engineering — is navigate the layers of chaos that real enterprise data contains.

DAB captures four hard requirements that expose this:

Multi-database integration. A single business question might require data from a transaction database and a CRM and a support ticketing system. Each speaks a different query dialect. The agent must route sub-queries correctly, translate between systems, and merge results without silently dropping data.

Ill-formatted join keys. Customer IDs in PostgreSQL might be integers. The same customers in MongoDB might appear as CUST-00123 strings. Nobody documented this inconsistency. The agent must detect it, resolve it, and join correctly — without being told there's a problem.

Unstructured text transformation. Some queries require extracting structured facts from free-text fields before any calculation is possible. “How many support tickets had negative sentiment?” requires reading the notes, extracting sentiment, then counting. Skip the extraction step and you return garbage.

Domain knowledge gaps. The schema says nothing about what “active customer” means in this industry — is it anyone in the database? Anyone who bought in the last 90 days? The answer matters. An agent without domain knowledge will pick the convenient interpretation, produce a confident result, and be wrong.

None of these are problems you can solve by switching to a better model. They’re engineering problems.

What “Context Is the Bottleneck” Actually Means

When we studied the architecture of Claude Code — Anthropic’s autonomous coding agent, whose source code was accidentally published to npm in early 2026 — one insight kept surfacing. The same insight appears in OpenAI’s writeup of their internal data agent.

The bottleneck in production AI agents is not generation. It’s context.

An agent that can’t find the right table across multiple databases will fail. An agent that doesn’t know what “revenue” means in this organization’s data will fail. An agent that forgets what the user corrected it on in the last session will keep making the same mistake.

Claude Code addresses this with a three-layer memory architecture: a persistent index file that loads at every session start, topic-specific files loaded on demand, and searchable session transcripts. OpenAI’s internal data agent uses six context layers. The specific numbers differ; the principle is the same. You have to engineer what the agent knows before the question arrives.

For our agent, this meant building a Knowledge Base — structured documents injected directly into the agent’s context window at session start. Not summaries. Not embeddings. Actual documents the agent can read and reason from.

The discipline here is counterintuitive: it’s about removal, not accumulation. A knowledge base that grows without being tested becomes noise. Every document we added, we tested by injecting it into a fresh context and asking a question it should answer. If the agent couldn’t answer correctly from that document alone, the document got revised or cut.

The result was three layers of knowledge: architecture patterns we extracted from the Claude Code source analysis, domain knowledge about the DAB datasets (how join keys were formatted, what status codes meant, which fields contained unstructured text), and a running corrections log — every time the agent failed on a query, we documented what went wrong and what the correct approach was, in a structured format the agent could read at the start of the next session.

That corrections log is what makes the whole system compound. The agent doesn’t improve because the model changes. It improves because the context it has access to improves.

The Self-Correction Problem

One of the hardest engineering challenges wasn’t getting the agent to answer correctly on the first try. It was getting it to recover when it didn’t.

An agent that surfaces a database error to the user isn’t a product. An agent that silently returns a wrong answer is worse. What you need is an agent that detects its own failures, diagnoses where in the execution chain something went wrong, and tries a different approach — without the user ever seeing the machinery.

This requires something more than a retry loop. The agent needs to distinguish between different types of failure:

• Is the query syntactically wrong?
• Is it semantically wrong (querying the right database with the wrong logic)?
• Is it a join key format mismatch?
• Is it a data quality issue in a specific field?

Each failure type has a different recovery path. A query that fails because of a join key format mismatch needs a resolver, not a rewrite. A semantic failure needs the agent to revisit its understanding of the question. Getting this right required building an adversarial probe library — a structured set of queries designed to expose specific failure modes — and iterating on the recovery logic until the agent handled each category reliably.

What “Evaluation Harness” Really Means

Here’s a thing that sounds obvious but often isn’t implemented: you cannot tell if you’re improving unless you have a measurement system that you trust.

This means more than running some test queries. It means:

• Tracing every tool call the agent makes on every query
• Recording query outcomes against known expected results
• Detecting regressions — cases where a change that fixed one query broke another
• Producing a score that reflects actual performance, not cherry-picked wins

Without this, you’re flying blind. You make a change, the demo looks better, and you think you’ve improved. Maybe you have. Maybe you’ve improved performance on two queries and broken four others that you didn’t check.

Our harness produced a score log across the full run. Not just the final number — a progression showing the baseline score, each change, and its impact. The benchmark submission we made to the DAB repository contained that log alongside the results JSON. The harness is the real deliverable, not the final score.

The Compound Effect Nobody Teaches

Something happens when a team builds a system like this with genuine role separation and genuine knowledge-sharing.

The engineers building the agent tell the knowledge team every time the agent fails. The knowledge team updates the corrections log. The agent’s context improves. The agent performs better. The engineers notice the agent is using the new knowledge correctly. The cycle compounds.

Meanwhile, someone is documenting the process in public — writing about what’s actually being learned, not just what’s being shipped. That documentation attracts attention from practitioners working on similar problems. Sometimes that attention brings back technical insight the team wouldn’t have found on its own.

This is what “compound engineering” means in practice. It’s not a metaphor. It’s an architecture pattern for how a team works, and it produces outcomes that no individual working in isolation can match in the same timeframe.

The Honest Part

The 54.3% benchmark ceiling isn’t just a challenge to beat. It’s a useful lens on the state of the field.

Most AI demos work because they’re run on prepared inputs, by people who know which questions to ask. Production deployments don’t have that luxury. Enterprise data is inconsistent, undocumented, and full of implicit knowledge that nobody ever wrote down. The gap between a demo that works and a system that works reliably at scale is almost entirely an engineering problem, not a model capability problem.

Closing that gap requires building context architecture before writing queries. It requires harnesses that measure regression, not just progress. It requires systematic failure analysis rather than anecdotal testing. It requires the kind of documentation discipline that makes a system recoverable if the codebase were deleted tomorrow.

None of this is glamorous. None of it shows up in a benchmark score headline. All of it is what separates systems that survive contact with real data from ones that don’t.

This post is based on work done during Weeks 8–9 of the TRP1 Forward-Deployed Engineering programme, building on the DataAgentBench benchmark from UC Berkeley’s EPIC Data Lab. The DAB paper is available at arxiv.org/html/2603.20576.