Evaluating Context Engineering for AI Agents: How to Measure What the Model Sees
Testing Write, Select, Compress, and Isolate in the Claude Agent SDK Deep Search Agent
Evaluating Context Engineering for AI Agents: How to Measure What the Model Sees
Testing Write, Select, Compress, and Isolate in the Claude Agent SDK Deep Search Agent
An agent can return a correct report while carrying a poor context architecture. It may retrieve fifteen documents when two contain the necessary evidence, expose dozens of irrelevant tools, preserve a stale memory, keep old tool output active, or send a subagent the parent agent’s entire trajectory. A final-answer grader can mark the run successful without detecting any of those problems.
**Part 1 introduced Write, Select, Compress, and Isolate as four operations for controlling context. Part 2 implemented them in a Claude Agent SDK research agent. This final part evaluates those operations directly. The central question is narrower than generic agent evaluation: how do we know that the context given to an agent is actually the right context?**
We will use the same research-agent codebase and preserve **Part 2’s open-ended task**, “Compare the main approaches for long-term memory in production AI agents and explain their trade-offs,” as the end-to-end regression. Controlled fixtures then test individual context policies with known relevant facts, distractors, contradictions, and expected handoffs. This two-level design is necessary because a broad report can reveal whether the system finished, but it cannot identify why a retrieval or memory policy worked.
Table of Contents:
- Why Outcome Success Is Not Enough
- Instrument the Context Pipeline
- Evaluate Write: Did We Preserve What Matters?
- Evaluate Select: Retrieval, Evidence Use, and Tool Exposure
- Evaluate Compress: Did Necessary Information Survive?
- Evaluate Isolate: Keep Unnecessary State Outside
- Stress-Test the Four Context Failure Modes
- Run Context Ablations
- Build a Scorecard and Optimize the Policy
- From Naive to Engineered to Evaluated
My new live cohort: **Claude Agent Engineering**. Six 3-hour live sessions, six office hours, and one agent system you build layer by layer.
There are only 20 seats, starting 1 November. Founding rate: $320 instead of $400 until Sunday, 27 September, 23:59 Helsinki time. Use code CLAUDECOURSE20
1. Why Outcome Success Is Not Enough
Part 2 already gives us a concrete warning. Both the naive and context-engineered research agents produced acceptable artifacts in all three runs. If we measure only task completion, the two systems appear equal:
# measurements/part2-agent-comparison.json
Naive Context-engineered
Artifact success 3 / 3 3 / 3
Mean final context 17,184 32,668 tokens
Mean estimated cost $0.663 $3.270
SDK success result 3 / 3 1 / 3The engineered version performed more work: it spawned three subagents, wrote workspace artifacts, called more tools, and sometimes needed another query to complete the report contract. Its mean local cost estimate was 4.93 times the naive estimate, and two runs ended with an SDK budget subtype even though the final report artifact passed. These measurements do not prove that context engineering is harmful. They prove that adding memory, retrieval, and subagents does not automatically create an efficient architecture.
Outcome quality asks whether the final report is correct, complete, and properly cited. Context quality inspects the pipeline that produced it. For each model decision, good active context has five properties:
- Relevant: each item is related to the current decision.
- Sufficient: the necessary evidence and constraints are present.
- Correct: supplied information is accurate and current enough for the task.
- Consistent: contradictions are removed, resolved, or made explicit.
- Efficient: irrelevant tokens do not substantially exceed the information required.
These properties interact. A context containing one correct fact may be precise but insufficient. A context containing every available document may have perfect recall but poor precision and unnecessary cost. The evaluation target is therefore not “smallest context.” It is the smallest context that preserves the evidence required for the next decision.
The separation between context precision and recall also appears in the **RAGAS evaluation framework. More broadly, the [Lost in the Middle study](https://arxiv.org/abs/2307.03172)** found that access to longer context does not make the position of relevant evidence irrelevant. Both results support measuring context composition rather than treating capacity as context quality.
Figure 1 — Outcome evaluation inspects the final answer. Context evaluation inspects which information moved from the available environment into the model’s active context before that answer.
A correct answer is necessary evidence of agent quality, but it is not evidence that the context pipeline was selective, robust, or economical.
2. Instrument the Context Pipeline
An evaluation harness needs more than application logs. “Called search_memory” tells us which action occurred, but not which instructions, schemas, memories, knowledge chunks, tool results, or history were active when Claude selected it. We need a context trace for each trial.
**Anthropic’s agent-evaluation guidance** separates a task, repeated trials, graders, a transcript, and the outcome. The transcript is particularly important for agents because two runs can reach the same answer by different tool trajectories. Our trace adds context-specific fields to that structure:
# context_eval/runtime.py
async def run_client_query(*, prompt, options, trace, snapshot_label="final"):
async with ClaudeSDKClient(options=options) as client:
await client.query(prompt)
async for message in client.receive_response():
trace.observe(message)
if snapshot_label:
trace.observe_context(
snapshot_label,
await client.get_context_usage(),
)A snapshot separates system-prompt, system-tool, MCP-tool, memory-file, message, and free-space tokens. That breakdown lets us test a claim such as “tool search reduced schema exposure” instead of inferring it from a lower bill. We use the SDK’s structured-output contract because deterministic graders are more reliable when they compare evidence IDs and expected actions rather than parse prose.
The trace is not a prompt dump for permanent retention. Raw traces can contain user data, tool payloads, and secrets, so a production implementation needs access controls, redaction, and a retention policy. The publishable dataset removes the SDK’s visual grid representation from snapshots but preserves the token categories and scored outputs needed to reproduce every result in this article.
Figure 2. A context trace joins what the model received, what it did, and what the run produced. This makes the context policy observable rather than inferred from the final answer.
3. Evaluate Write: Did We Preserve What Matters?
Write decides what becomes durable state. The test must penalize two different failures: storing noise and failing to store a fact needed later. Those are write precision and write recall.
The fixture contains ten observations: three verified durable facts, three temporary facts, three irrelevant observations, and one contradicted statement. The model must return IDs for the items it would persist. A code-based grader compares that set with the three gold IDs:
# context_eval/scoring.py
def selection_metrics(selected, gold):
selected_ids = set(selected)
gold_ids = set(gold)
true_positive = len(selected_ids & gold_ids)
false_positive = len(selected_ids - gold_ids)
false_negative = len(gold_ids - selected_ids)
precision = true_positive / len(selected_ids) if selected_ids else 0.0
recall = true_positive / len(gold_ids) if gold_ids else 1.0
return {
"precision": precision,
"recall": recall,
"false_positive": false_positive,
"false_negative": false_negative,
}The same function is useful for writing and retrieval because both operations select a subset from an available collection. Their interpretation differs: a false positive in Write can become durable context poisoning, while a false negative means a later step may have no way to recover a required decision.
Across two live trials, the agent selected all three durable facts and rejected all seven other observations. Write precision and recall were both 100%, and the contradicted item was never persisted. This is a passing fixture, not proof that arbitrary memory writing is solved. The realistic next step is to expand the fixture with paraphrases, partially verified claims, changing facts, and adversarial instructions, then repeat each case enough times to estimate variance.
Figure 3. Write precision measures whether stored items deserved persistence; write recall measures whether every durable fact crossed into the scratchpad or memory store.
Evaluate the write decision when it occurs. Waiting until a stale or unsupported memory causes a future failure makes the source of the error harder to identify.
4. Evaluate Select: Retrieval, Evidence Use, and Tool Exposure
Selection operates over several collections. The research agent selects scratchpad files, memories, knowledge chunks, and tool schemas. One aggregate “retrieval score” hides which collection is failing, so the harness gives each boundary its own gold set.
The memory fixture resembles a project store contaminated with personal preferences and unrelated incidents. The task is to diagnose a Python authentication failure. Three memories matter: the runtime version, the OAuth2 architecture, and a project-specific authentication constraint. The retriever is required to return five items. Knowledge retrieval has two required documents and a top_k of four.
# context_eval/experiments.py
memory_score = selection_metrics(
memory_ids,
fixture["gold_memory_ids"],
)
knowledge_score = selection_metrics(
knowledge_ids,
fixture["gold_knowledge_ids"],
)
# Grade what the model cited separately from what retrieval returned.
memory_use = selection_metrics(
output["memory_evidence_ids"],
fixture["gold_memory_ids"],
)Both trials returned every required item, so memory and knowledge recall were 100%. Precision was only 60% for memory and 50% for knowledge because the fixed result limits also admitted two distractors from each collection. This is a useful diagnosis: increasing top_k protected recall, but active context carried avoidable noise.
Retrieval and evidence use must remain separate. In the first trial, the structured answer cited two of the three required memories and both knowledge documents, with no irrelevant citations. In the second, the model returned a diagnosis but left both evidence arrays empty. The retriever behaved identically in both trials; the difference occurred after selection. A retrieval-only dashboard would miss it.
Tool selection needs the same treatment. The harness exposes a catalog of 50 in-process MCP tools, with three required to diagnose a redirect mismatch. One configuration sets ENABLE_TOOL_SEARCH=false; the other sets it to true:
# context_eval/experiments.py
options = base_options(
allowed_tools=["mcp__catalog__*"],
mcp_servers={"catalog": server},
env={"ENABLE_TOOL_SEARCH": "true" if tool_search else "false"},
)Both single trials selected exactly the three required tools. More importantly, both final context snapshots reported 3,159 MCP-tool tokens. The enabled run had a lower local cost estimate ($0.020 versus $0.047), but identical schema exposure means we cannot attribute that difference to tool deferral.
The current SDK tool-search documentation explains that tool search can defer large catalogs and warns that large tool surfaces consume context; our measurement shows why the implementation must verify activation in its own trace. A feature flag is not evidence that selection occurred.
The appropriate optimization follows directly from the failure. Memory and knowledge retrieval need tighter ranking or smaller adaptive limits while protecting recall. The in-process catalog needs configuration or a catalog size that actually activates schema deferral before tool-search savings can be claimed. In both cases, rerun the same fixtures after changing the policy.
Figure 4. Selection recall asks whether every required item was entered into the context. Precision asks how much irrelevant material accompanied it. Tool-schema exposure must be measured from the context snapshot.
Measure retrieval, context admission, evidence use, and final correctness separately. They are adjacent stages, not interchangeable metrics
5. Evaluate Compress: Did Necessary Information Survive?
Compression is successful only when later decisions still have the information they require. A smaller context window is an intermediate measurement, not the objective.
The controlled fixture plants seven facts that should survive: the user goal, a confirmed decision, an implementation constraint, a blocker, a source URL, a failed approach, and the current plan. It also plants three nonce-marked disposable facts representing old terminal output, an unrelated search result, and a temporary metric. CLAUDE.md tells the compactor which categories to preserve and discard.
# runs/compression-auto/CLAUDE.md
When compacting, preserve the original goal, confirmed decision, implementation
constraint, current blocker, source URL, failed approach, and current plan.
Discard repeated raw output, unrelated search results, and temporary metrics.The SDK agent-loop guide documents automatic compaction and the compact_boundary system event. The first manual test was invalid: it had only 29,024 active tokens, /compact emitted no compact_boundary, and the before and after snapshots were identical. The harness now rejects that state. The final trial builds disposable history in bounded chunks until automatic compaction is observed:
# context_eval/experiments.py
for chunk in range(1, 9):
before = await client.get_context_usage()
await client.query(
"Append this disposable diagnostic history to the current session. "
"Reply only READY after reading it.\n\n"
+ _compression_noise(offset=chunk * 600)
)
async for message in client.receive_response():
trace.observe(message)
after = await client.get_context_usage()
if trace.compactions:
trace.observe_context("before_compaction", before)
trace.observe_context("after_compaction", after)
break
if not trace.compactions:
raise RuntimeError("The SDK did not emit a compact_boundary")After the boundary, a resumed structured-output query asks for the exact value of all ten IDs and requires NOT_RETAINED when a value is unavailable. The result was:
# python scripts/show_compression.py ../measurements/context-evaluation.json
Observed automatic compaction
Boundaries 1
Active tokens before 156,282
Active tokens after 30,111
Compression ratio 80.7%
Required facts retained 7/7
Disposable facts removed 3/3This trial achieved both sides of the objective: substantial reduction and complete retention on the planted probe. It was also the most expensive test in the suite, with a $6.578 SDK estimate, because reaching the real automatic boundary required repeatedly processing a growing history.
Production compression tests should run less frequently than small retrieval fixtures, but they cannot be replaced by a mocked summary if the goal is to validate the SDK boundary and session continuation.
Scratchpads and compaction still solve different problems. A scratchpad is curated application state written deliberately for later use. Compaction reduces accumulated conversational history.
A system can pass this retention probe and still have a poor scratchpad policy, or write perfect notes while losing an important conversation constraint during compaction.
Figure 5. The observed boundary removed 80.7% of active tokens while retaining all seven required probes and discarding all three nonce-marked temporary items.
6. Evaluate Isolate: Keep Unnecessary State Outside
Isolation asks how much the application knows compared with how much a particular model call receives. We test subagent handoffs and environment artifacts because both can hide large payloads behind a small interface.
Programmatically defined **Agent SDK subagents receive a scoped prompt, tools, and their own conversation before returning a final handoff. The parent context contains 30 records, but only E1 and E2 describe the registered and emitted OAuth callback. The lead agent must create one synchronous callback-verifier** task, pass only those records, and keep the delegated prompt under 500 characters:
# context_eval/experiments.py
agent = AgentDefinition(
description="Verifies one supplied OAuth callback claim.",
prompt=(
"Use only evidence passed in the delegated prompt. Return a concise "
"finding, evidence IDs, and one limitation."
),
tools=[],
model=model,
)The actual delegated prompt was 150 characters, included both required IDs, and included none of the 28 unrelated parent records. Only the subagent’s concise conclusion returned to the lead. The trial also wrote a 180,000-row CSV of 1,670,340 bytes to the execution environment and exposed a 114-character metadata summary. The artifact exposure ratio was therefore 0.0068% by characters-to-bytes proxy.
That proxy is deliberately simple. Bytes and model tokens are not equivalent units, and a real multimodal artifact may require a different denominator. The invariant is more important than the unit: the full artifact must remain addressable in the environment, while the model receives only the statistics, ranges, or excerpts needed for its current decision.
The subagent trace also revealed a measurement limitation. Although the SDK’s **cost-tracking guide describes usage accounting, the message stream in this OpenRouter-backed trial recorded one Agent** call but did not provide separately attributed subagent token usage. We can grade handoff content and count the call, but should not claim a precise subagent-token saving from this trial.
Figure 6. The application retained 30 parent records and a 1.67 MB artifact, while the subagent received two evidence records and the lead received a concise result.
Exposure ratio is a boundary metric: state can remain available to the application without becoming active model context.
7. Stress-Test the Four Context Failure Modes
Normal examples often underrepresent the poisoning, distraction, confusion, and clash patterns described in **Anthropic’s context-engineering guidance**. A stress suite should inject them deliberately and grade both the answer and the corrective action.
The fixture covers poisoning, distraction, and clash directly. The poisoning case supplies a stale memory claiming PostgreSQL 14 and current environment evidence showing PostgreSQL 16; the expected action is to verify and replace the stale memory.
The distraction case surrounds the current error with repeated failed attempts; the agent must choose a new strategy. The clash case supplies old Python 3.10 state and a current Python 3.12 inspection; the newer observation must supersede the old one. Context confusion is tested separately by the 50-tool catalog containing three useful and 47 irrelevant tools.
# context_eval/scoring.py
answer_ok = fact_matches(result["answer"], case["expected_answer"])
action_ok = result["action"] == case["expected_action"]
passed = answer_ok and action_okAll three structured stress cases passed in both repetitions, producing six passes from six probes. Both tool-catalog variants also called exactly the three useful tools and none of the 47 distractors.
These are controlled pass results, not estimates of a general failure rate. Stronger poisoning tests should vary source authority and recency; distraction tests should change the wording of failed attempts; confusion tests should include tools with overlapping names and descriptions.
This suite also demonstrates why the expected action matters. Returning “PostgreSQL 16” answers the immediate question, but leaving the PostgreSQL 14 memory untouched lets the same poison re-enter later contexts. A robust agent must identify the correct fact and repair or quarantine the context source that supplied the bad one.
Figure 7. Context stress tests make failure modes repeatable by planting a wrong fact, a repeated failed history, irrelevant options, or conflicting versions before grading the response and remediation.
8. Run Context Ablations
Ablations answer a causal question that dashboards cannot: does a particular context component help on this task? We run one labeled OAuth diagnosis with eight context configurations. Each configuration sees a known subset of 14 available records, and the grader checks the root cause, proposed fix, rejected wildcard fix, and three required evidence IDs.
The naive configuration receives memory, knowledge, the current tool result, and four old attempts. The context_engineered configuration replaces old history with scratchpad and subagent records but still exposes distractors. The evaluated configuration admits only six records already demonstrated useful by the trace. Five more variants remove one context operation at a time.
# context_eval/experiments.py
if variant == "no_memory":
groups.remove("memory")
elif variant == "no_tool_retrieval":
groups.remove("tool_results")
elif variant == "no_scratchpad":
groups.remove("scratchpad")
elif variant == "no_compaction":
groups.append("uncompacted_history")
elif variant == "no_subagents":
groups.remove("subagent_handoff")Two trials per configuration produced the following result:
# python scripts/show_ablations.py ../measurements/context-evaluation.json
Configuration Success Precision Exposed Context
naive 100% 27% 79% 1,525
context_engineered 100% 60% 71% 1,486
evaluated 100% 100% 43% 1,329
no_memory 0% 71% 50% 1,366
no_tool_retrieval 50% 56% 64% 1,433
no_scratchpad 100% 50% 57% 1,390
no_compaction 100% 43% 100% 1,677
no_subagents 100% 56% 64% 1,430The first three versions all solved this small task, but their context precision increased from 27% to 60% to 100%. Removing memory caused both trials to fail because the registered callback was missing. Removing the current tool result reduced success to one of two because the emitted URI was missing. Removing the scratchpad, compacted-history policy, or subagent handoff did not reduce task success in two trials, but it did reduce context precision or increase exposure.
This is the correct interpretation of a small ablation: memory and the current runtime result were necessary for this fixture; the other components were redundant for immediate task success under these conditions. It does not establish that scratchpads or subagents are unnecessary for long research tasks. It tells us which mechanisms earn their complexity on this test and which need a different fixture to demonstrate value.
Figure 8. Ablations hold the task and grader constant while changing one context component, so task success can be interpreted alongside precision and exposure.
9. Build a Scorecard and Optimize the Policy
The scorecard should preserve the structure of the context architecture. A single weighted number can be useful for regression gates, but it hides whether a failure originated in writing, selection, compression, or isolation. The primary report therefore keeps one question and one metric family for each dimension:
- Write: Did the system save what became important? Measure write precision, recall, and incorrect writes.
- Select: Did the current step receive the required subset? Measure retrieval precision, recall, evidence use, and schema tokens.
- Compress: Did required information survive while disposable history disappeared? Measure retention, discard, and compression ratio.
- Isolate: Did unrelated state remain outside the call? Measure exposure ratio and handoff precision.
- Correctness: Was supplied context trustworthy and conflict-aware? Measure incorrect-context rate and stress-case passes.
- Efficiency: How much context and money did the run consume? Measure active tokens, cumulative input, latency, and estimated cost.
- Outcome: Did the agent complete the task? Measure task success, artifact checks, citations, and any domain-specific grader.
The captured scorecard is intentionally uneven. It shows strong Write and compression probes, high-recall but low-precision retrieval, and a tool-search configuration that did not change schema exposure:
# python scripts/show_scorecard.py ../measurements/context-evaluation.json
Context evaluation scorecard
Write precision 100.0%
Write recall 100.0%
Memory select P / R 60.0% / 100.0%
Knowledge select P / R 50.0% / 100.0%
Compression retention 100.0%
Compression discard 100.0%
Context failure pass rate 100.0%
Included SDK estimate $7.349Optimization now becomes a traceable engineering loop. The retrieval result suggests reducing or adapting top_k, improving ranking, then verifying that recall remains 100%. The tool result requires fixing or verifying deferred loading before expecting schema-token savings.
The compression result passes its planted probe, so the next test should increase semantic difficulty rather than merely add more repeated noise. The isolation test should add reliable subagent usage accounting before making a cost claim.
This order matters: trace, evaluate, identify the failing boundary, change one policy, and rerun the same fixture. Optimizing token totals first can produce a smaller context that omits necessary evidence. Outcome quality remains the guardrail while context metrics explain how the system reached it.
Figure 9. Context optimization begins with a trace, changes one policy in response to a measured failure, and reruns the same fixture while retaining task success as a guardrail.
Do not optimize context size in isolation. Precision, sufficiency, correctness, and outcome must remain visible beside token and cost measurements.
10. From Naive to Engineered to Evaluated
The three versions now represent different levels of control. The naive agent lets available information accumulate. The context-engineered agent implements Write, Select, Compress, and Isolate. The evaluated agent keeps those mechanisms but tunes admission policies against labeled traces.
On the compact ablation task, all three returned the correct fix in both trials. The progression appeared in the context rather than the outcome: precision improved from 27% to 60% to 100%, exposure fell from 79% to 71% to 43%, and mean final active context fell from 1,525 to 1,486 to 1,329 tokens. The differences are small because the fixture is small, but the direction is measurable, and the required evidence recall stayed at 100%.
Part 2’s open-ended research runs prevent us from overstating that result. There, the engineered architecture cost more and used a larger final context than the naive baseline, even though both produced acceptable reports.
Part 3 explains how to respond: inspect which component expanded the trajectory, isolate it with an ablation, and keep it only if the task or context score improves. Architectural sophistication is a hypothesis until a controlled evaluation supports it.
The current evidence has clear limits. Write, Select, and stress tests have two repetitions; tool search, compression, and isolation have one. The ablation fixture is a compact OAuth diagnosis, not the full research workload. Some graders use exact evidence IDs and normalized string matching, which makes them reproducible but narrow.
SDK cost values are estimates, and subagent tokens were not separately attributed in the isolation trace. A production suite should add more tasks, more trials, latency distributions, human review for open-ended reports, and regression thresholds based on observed variance.
The series now forms one engineering progression. Part 1 established how to think about context. Part 2 moved working state into files, selected tools and knowledge, compacted history, and isolated subagents.
Part 3 made those policies observable and testable. Memory, RAG, compaction, and subagents are not the end of context engineering; they are decisions about what the model sees, and those decisions require evidence.
The governing principle remains unchanged: the goal is not to give the model more context. It is to give it the right context for the next decision, then measure whether that happened.
Figure 10. The series progresses from understanding context to engineering its boundaries to measuring whether each boundary supplies the right information at the right time.
My new live cohort: **Claude Agent Engineering**. Six 3-hour live sessions, six office hours, and one agent system you build layer by layer.
There are only 20 seats, starting 1 November. Founding rate: $320 instead of $400 until Sunday, 27 September, 23:59 Helsinki time. Use code CLAUDECOURSE20
메타데이터
- post_id
- 0dc6d2a94be0
- slug
- evaluating-context-engineering-for-ai-agents-how-to-measure-what-the-model-sees-0dc6d2a94be0
- url
- https://levelup.gitconnected.com/evaluating-context-engineering-for-ai-agents-how-to-measure-what-the-model-sees-0dc6d2a94be0
- canonical_url
- https://levelup.gitconnected.com/evaluating-context-engineering-for-ai-agents-how-to-measure-what-the-model-sees-0dc6d2a94be0
- author_url
- https://medium.com/@yousefhosni
- status
- ok
- fetched_at
- 2026-09-06 23:22:30
