Why Bigger Context Windows Still Don’t Fix AI Agents

Why reliability breaks long before raw model intelligence does

Photo by Immo Wegmann on Unsplash

When an agent fails, we usually reach for the same explanation first: the model was not smart enough.

Its reasoning broke. The task was too complex. The prompt needed more work. The model simply was not capable enough.

That explanation is sometimes correct. But it is not the one I trust by default anymore.

After building and debugging more agent-style workflows, I started noticing a different pattern: many failures were not really intelligence failures. They were context failures.

Public postmortems point in the same direction. In Anthropic’s write-up on Claude Code quality regressions, users described the agent as forgetful, repetitive, and prone to strange tool choices. What matters is not just the symptom list. It is the diagnosis. The root causes were framed around reasoning effort, reasoning retention, and prompt behavior that degraded the agent’s working context. The model still had capability. What broke was the continuity of the information it needed to stay coherent.

That changed how I think about context engineering. I no longer see it as an extension of prompt engineering. I see it as a systems problem: how information gets selected, compressed, ordered, persisted, and reintroduced over time.

Prompt engineering helps a turn. Context engineering determines whether an agent can survive a workflow.

Context engineering is what turns a model into a reliable system

In a normal chat interaction, the model sees a system prompt, a user message, and perhaps a bit of recent history. In an agent, that is not the real environment.

An agent accumulates instructions, retrieved passages, tool outputs, working notes, failed attempts, summaries, partial plans, and execution state from earlier turns. Every tool call changes the next prompt. Every long result competes with the actual question. Every extra token can weaken the importance of something else.

Photo by Femke Plantinga on weaviate.io

That is why I think of context engineering as information logistics.

The real question is not just “what should I tell the model?” It is “what should the model see right now, what should stay outside the active window, and what should return only in compressed form?”

Once I started thinking about agent design that way, a lot of “random model behavior” stopped looking random.

LangChain puts this plainly in its own context-engineering docs: agents usually fail for one of two reasons, either the model is not capable enough or the right context was not passed in. More often than not, it argues, the second one is the real problem. That matches my experience, and it also matches the stronger public postmortems I trust.

The failure patterns look different, but they come from the same root problem

Longer context windows help, but they do not remove the problem. In some cases, they just make the breakdown slower and harder to diagnose.

These are the four patterns I keep seeing.

1. Accuracy collapse

The model remembers the topic, but not the relationship between facts. It keeps the noun and loses the dependency.

In practice, this looks like the system retaining surface familiarity while losing the operational detail that matters. Chroma’s context-rot work is useful here because it shows that even when task difficulty is held constant, reliability still degrades as inputs get longer. In real agent systems, that often surfaces as remembering the file or concept, but not the dependency chain or state transition that actually determines the next correct action.

Photo on Anthropic Blog

2. Instruction drift

The rule is still technically in the context, but it is no longer functionally active.

Formatting changes. Tone slips. Safety constraints weaken. The agent starts acting as though an earlier instruction no longer exists, even though it has not actually been removed.

3. Recursive failure

This is one of the most expensive ones. The agent retries the same bad action because the earlier failure is now buried under fresh output.

Anthropic’s Claude Code postmortem described this pattern in the wild: after prior reasoning was dropped under a broken cache path, the agent would continue operating but increasingly without memory of why it had made earlier edits and tool calls. Users experienced that as repetition and strange tool behavior. That is a good example of recursive failure in practice. Once the system loses the compact explanation of what just happened, it becomes much easier to repeat the same mistake.

4. Flaky long-run behavior

The workflow works in short runs and fails in longer ones. Those are the bugs that feel random until you realize the context itself is degrading.

“Context rot” is still the best phrase I have seen for this. The information is present, but the agent stops using it reliably.

The Chroma report is especially valuable because it isolates input length itself as a variable. Its results suggest that longer windows are not a free reliability upgrade. Even simple tasks can become less dependable as more tokens accumulate, which is exactly why “just give the agent more context” is often the wrong design instinct.

Rule 1: Compress the representation, not the meaning

When context grows too large, truncation is the usual reflex.

Sometimes that is necessary. But I do not think it should be the first move.

Blind truncation is cheap, but it often removes the one detail that made the next step possible.

What works better for me is layered compression.

If a tool returns a huge payload, I usually do not want the payload in the next turn. I want the conclusion the payload supports.

If a user lookup returns a 2,000-line structure and the relevant fact is simply that “Alice exists and her ID is 1,” then that is probably what belongs in the active context. Not the headers. Not unused metadata. Not the raw object just because we technically have room for it.

The same rule applies to history.

Recent turns often deserve to stay verbatim. Older turns usually deserve summaries. Not because they are unimportant, but because their role has changed. They are no longer live interaction. They are state.

Once something becomes state rather than dialogue, it should stop consuming premium attention.

Rule 2: External memory is part of the architecture

One of the biggest improvements in my own workflows came from stopping the assumption that everything must live inside the prompt window.

It should not.

For long-running agent tasks, I now prefer to keep a lightweight state file or scratchpad with items like:

• confirmed facts
• failed attempts
• current hypothesis
• next action
• unresolved questions

That approach works much better than dragging the full execution history forward turn after turn.

This is not just a private preference among builders. Anthropic explicitly recommends structured note-taking for long-horizon tasks and points to examples like Claude Code maintaining to-do lists or Claude playing Pokemon by reading and updating notes across resets. That matters because it reframes external memory as part of the architecture, not as a hack around missing model capability. The benefit is not just lower token pressure. The behavior becomes easier to debug because the evolving state is visible instead of implicit.

I increasingly think of file-backed working state as part of context engineering itself, not as a workaround around it.

Photo on Manus Blog

Rule 3: Order context like a pipeline, not a dump

Context quality is not only about what gets included. It is also about where things appear.

Important material in the middle of a long prompt is easier to lose than people expect. So I now bias toward a predictable order:

1. system-level rules
2. durable user or task memory
3. current objective
4. relevant retrieved knowledge
5. refined tool outputs
6. working summary and output constraints

That order is not sacred. But randomness is expensive.

I also prefer explicit structural boundaries between instructions, evidence, tool outputs, and the current task. Not because XML-style tags are fashionable, but because they reduce ambiguity and make failures diagnosable.

A giant undifferentiated blob is hard for the model to use and even harder for humans to debug.

Better agents will come from better context routing, not just bigger models

The practical lesson I keep returning to is simple: many agent failures are context-routing failures.

We gave the model too much of the wrong thing, too little of the right thing, or the right thing in the wrong form. We kept stale outputs hot. We kept history that should have become state. We preserved verbosity when what the model actually needed was salience.

That is why I think context engineering is becoming one of the real engineering disciplines around agents.

Stronger models may tolerate more mess, but long-running agents will still live under attention pressure. Someone still has to decide what stays active, what gets summarized, what moves to external memory, and what should never have entered the working set in the first place.

That work is not prompt polish. It is system design.

And in my experience, it is often the difference between an agent that looks impressive in a demo and one that remains reliable after the tenth tool call.

What I check before I trust an agent workflow

Before I trust an agent workflow, I now ask:

• What information must stay verbatim?
• What information should become summary state?
• What tool outputs are too verbose to keep raw?
• What can move into external memory?
• What failure should the agent never be allowed to forget?

Those questions have improved my agent systems more than simply making the context window larger.

A tiny state file like this is often enough to keep an agent from losing the thread:

# Working State

## Confirmed Facts
- auth.py fails because the client library version changed
- The retry loop already hit the same endpoint twice

## Next Step
- Inspect token refresh flow in auth.py

If there is one claim I would keep from all of this, it is this: the next leap in agent reliability will not come from context windows alone. It will come from treating context as a designed system with budgets, structure, memory, and failure recovery.

References

• Anthropic — An Update on Recent Claude Code Quality Reports
• Anthropic — Effective Context Engineering for AI Agents
• LangChain Docs — Context Engineering
• Chroma Research — Context Rot