Recursive Language Models: A Different Approach to Long Context

Previous posts explained why bigger context windows don’t fix reasoning. Context rot is structural — attention dilutes, positions extrapolate poorly, ambiguities compound.

MIT’s Recursive Language Models (RLMs) solve this by rejecting a fundamental premise: context must be tokens in a prompt.

Instead, RLMs treat context as data in an external environment. The model becomes an orchestrator that decides how to decompose and interact with context, rather than a processor forced to attend across it uniformly.

The results are:

• RLM(GPT-5-mini) achieves 2× better performance than GPT-5 on long-context tasks
• On BrowseComp-Plus with 1000 documents, RLM maintains 95% accuracy while baselines collapse to 55%
• This works with frozen models — no retraining required

Striking ain’t it?

This post explains how RLMs work, why they avoid context rot, and what the architecture actually looks like.

The Core Insight: Context as Environment

Traditional LLM usage:

The LLM receives both the query and all context tokens. Attention dilutes across everything.

RLM approach:

The Root LM receives only the query. Context lives in an external environment. The Root LM generates code to explore and process context.

This single change eliminates attention dilution. The Root LM’s attention focuses on the query and its own reasoning, not on competing context tokens.

Architecture: Four Components

RLMs consist of four components working together.

1. Root LM (Depth=0)

The primary decision-maker. Crucially, it receives only the user’s query, not the context tokens.

For example, on OOLONG benchmark with 3000–6000 rows of data, the Root LM sees only:

“Among instances for user IDs {67144, 53321, …}, how many should be labeled as ‘entity’?”

No context tokens. No attention dilution. The prompt remains small and computationally clean.

2. REPL Environment

A Python REPL (like a Jupyter Notebook) with the full context preloaded as a variable named context.

The Root LM interacts with this environment by generating Python code:

# What the Root LM generates
lines = context.split('\n')
filtered = [l for l in lines if 'user_id' in l and any(uid in l for uid in [67144, 53321])]
print(f"Found {len(filtered)} matching lines")

The output returns to the Root LM. The full context remains in memory, not in the model’s prompt.

3. Recursive LMs (Depth=1)

For semantic tasks requiring understanding of context chunks, the Root LM invokes Recursive LMs from within the REPL:

# Recursive invocation
for chunk in filtered_chunks:
    sub_result = recursive_lm.call(
        query="Label this instance as 'entity' or 'other'",
        context=chunk  # Only the chunk, not full context
    )
    results.append(sub_result)

Recursive LMs are typically smaller models (e.g., GPT-5-mini) optimized for cost. Each processes only a small chunk, avoiding context rot even in sub-models.

4. Output Strategies

The Root LM returns answers in two ways:

Direct answer: FINAL("The count is 342")

Variable retrieval: FINAL_VAR(answer_list) — retrieves a Python variable constructed programmatically

The second strategy enables essentially unbounded output length. The model builds long outputs as Python variables, avoiding output token limits.

The Full RLM Loop

Here’s the complete execution flow:

Key properties:

• The Root LM never sees the full context
• Recursive LMs see only small chunks
• Each model call has a clean, focused prompt
• Decomposition emerges from the query, not predetermined steps

Why This Avoids Context Rot

1. Attention Remains Sharp

The Root LM’s attention focuses on:

• The user query
• Outputs from recursive calls
• Its own reasoning

No competing context tokens. No dilution.

2. Reasoning Is Controlled

The model explicitly decides which context to examine, at what granularity, through code execution. It doesn’t passively hope attention lands on the right tokens.

3. Decomposition Is Adaptive

Unlike fixed-prompt agents, decomposition strategy emerges from the model’s reading of the actual context. Two similar queries against different contexts may decompose differently.

4. Scaling Is Modular

No single model call handles the full context. If context is 1M tokens but each chunk is <50K, recursive LMs see only the chunk.

Benchmark Results: The Performance Gain

OOLONG: Distributional Reasoning

Task: Classify thousands of instances from 128K-263K token contexts and aggregate counts.

RLM(GPT-5-mini) outperforms GPT-5 while being cheaper per query. This reveals that context rot is more severe for smaller models — they lack capacity to manage attention across huge contexts.

BrowseComp-Plus: Multi-Document QA

Task: Answer questions requiring information from multiple documents.

RLM is the only approach able to maintain 95% accuracy at 1000 documents. Baselines collapse under the scale.

Emergent Strategies: What Models Discover

The most surprising finding: RLMs discover sophisticated strategies without explicit training.

These strategies emerge as the model learns it can write code to explore context.

1. Peeking

The Root LM prints the first 2000 characters to understand data format:

# Discovered strategy
print(context[:2000])
# Observes: rows separated by newlines, metadata fields
# Uses this schema for subsequent operations

2. Grepping

Rather than semantic retrieval, RLM discovers regex/keyword searches are more efficient for narrow patterns:

filtered = [l for l in lines if 'user_id' in l and uid in l]

3. Partition + Map

For semantic tasks, RLM partitions context into chunks and processes in parallel:

chunks = partition(context, chunk_size=1000)
for chunk in chunks:
    result = recursive_lm.call(query, chunk)

4. Summarization

RLM learns to summarize subsets for high-level decisions:

summary = recursive_lm.call("Summarize these 1000 rows", chunk)
# Root LM aggregates summaries

5. Programmatic Processing

For long-output tasks, RLM builds output as Python variables:

# Instead of generating 50K output tokens
results = []
for item in items:
    results.append(process(item))
FINAL_VAR(results)  # Return variable, not tokens

These strategies are not programmed. They emerge from giving the model tools (code execution, recursion) and letting it discover what works.

The Model to Carry Forward

RLMs avoid context rot by treating context as external data rather than prompt tokens. The Root LM sees only the query and orchestrates processing through code execution and recursive calls.

Attention remains sharp. Decomposition is adaptive. Performance improves.