Context Windows Are Lying to You: How LLMs Really Handle Long Inputs

“128k context” sounds like you can throw anything at it. Here’s why that’s dangerously wrong.

Photo by Omid Ziadzadeh on Unsplash

If you’ve read my earlier pieces on How Large Language Models Really Work (Part 1) and Part 2, you already know how tokens work, how sampling works, and how models generate text one token at a time.

Now let’s talk about something the marketing slides don’t explain, what actually happens when you use a large context window.

Because “128k context” or “1M context” sounds like a superpower. In practice, it’s more like a warning label.

The Promise vs. The Reality

OpenAI says GPT-4o supports 128k tokens. Anthropic says Claude supports 200k tokens. Google says Gemini 1.5 Pro supports 1 million tokens.

The implicit message: throw your entire codebase in there. Dump all your PDFs. Feed it your whole database. It can handle it.

And technically? It can. The model won’t crash. It won’t throw an error.

But handling and understanding are two completely different things.

What a Context Window Actually Is

At its core, the context window is the maximum number of tokens the model can “see” at once during a single forward pass. Everything you want the model to reason about including your system prompt, conversation history, retrieved documents, the user’s question must fit inside this window.

Think of it like RAM. Your laptop might have 32GB of RAM. That doesn’t mean every program runs equally well using all 32GB. There are performance cliffs, bottlenecks, and degradation points.

Context windows work the same way.

The Lost in the Middle Problem

This is the most well-documented failure mode of long-context LLMs, and it should fundamentally change how you use them.

Researchers at Stanford and UC Berkeley published a paper titled “Lost in the Middle: How Language Models Use Long Contexts” that showed something deeply inconvenient:

LLMs are dramatically better at using information that appears at the beginning or end of their context and significantly worse at using information buried in the middle.

They tested models by placing the answer to a question at different positions within a long context. Performance dropped sharply when the relevant information was in the middle, regardless of context window size.

Here’s what that curve looks like conceptually:

Performance
    │
100%│ ██                                      ██
 80%│   ██                                  ██
 60%│     ██                              ██
 40%│       ██                          ██
 20%│         ████████████████████████
    └─────────────────────────────────────────
    Start                                   End
                  Position in Context

The model is not reading your context like you read a book like linearly, with equal attention everywhere. It has positional biases baked in from training.

Practical implication: If you’re building a RAG system and stuffing 15 retrieved chunks into the context, the chunks in positions 5–10 are being read least carefully. Your most relevant chunk might be sitting in the dead zone.

Attention Doesn’t Scale Linearly

Here’s the deeper reason this happens and it’s architectural.

Transformers use self-attention, where every token attends to every other token. This is what makes them powerful. But the computational cost of attention grows quadratically with sequence length.

For a sequence of length N, attention requires O(N²) operations.

Double the context → 4x the computation. 10x the context → 100x the computation.

Model providers optimize around this (with techniques like sparse attention, sliding window attention, flash attention), but the fundamental tradeoff remains: longer context = more diffuse attention = harder to focus on what matters.

When you feed a model 100k tokens, its attention is spread across 100k tokens. Important signals get diluted. The model must work harder to surface what’s relevant and it doesn’t always succeed.

The Needle-in-a-Haystack Test

A common benchmark for long-context models is the “needle in a haystack” test. The setup:

• Take a large document (the “haystack”) — often Paul Graham essays or similar text
• Inject a single specific fact (the “needle”) at a known position
• Ask the model to retrieve that fact

Results vary a lot by model, but the pattern is consistent: retrieval accuracy degrades as context length grows, and degrades further when the needle is in the middle.

Some models like Claude 3 and Gemini 1.5 perform significantly better on this benchmark. But performing well on a synthetic benchmark is not the same as performing well on your messy, real-world data.

In production, your “haystack” is not clean prose. It’s PDFs with broken formatting, tables, footnotes, headers, and code all chunked imperfectly. The needle is not a neatly injected sentence. It’s a nuanced fact buried in a paragraph that the model might not even recognize as the answer.

Recency Bias: The Other Silent Problem

Beyond the lost-in-the-middle effect, there’s recency bias.

In long multi-turn conversations, LLMs tend to weight recent messages more heavily than earlier ones, even when the earlier context is more relevant.

You’ve probably experienced this. You have a 30-message conversation. The user references something they said in message 3. The model responds as if it doesn’t remember, or gets it wrong, even though it’s technically “in context.”

This isn’t forgetting the model can see all 30 messages. It’s that the attention mechanism, despite seeing the earlier content, assigns it lower effective weight by the time it’s generating a response.

Practical implication for chatbots and agents: Don’t assume that “in context” means “will be used correctly.” Critical instructions, constraints, and facts should be reinforced placed near the end of the context, or repeated in the system prompt.

The Tokenization Trap

Here’s something that catches engineers off guard when they first start measuring context usage seriously.

The relationship between text and tokens is not 1:1 and it’s not consistent.

• English prose: ~1 token per 0.75 words
• Code: often 1 token per 3–5 characters (more tokens per word due to symbols, indentation)
• Non-English languages: often 2–5x more tokens per word than English
• Structured data (JSON, XML): highly variable, nested JSON can tokenize very inefficiently

This means your “128k token context” is actually much smaller than it sounds for many real-world use cases.

Feed in a large JSON payload? You might burn 20k tokens before the model sees a single word of your actual question.

Feed in Python code? Indentation, brackets, and variable names tokenize expensive.

Feed in a document in Hindi or Arabic? Your effective context shrinks dramatically.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
english_text = "The model processes tokens sequentially during inference."
code_snippet = "def process_tokens(input_ids: List[int]) -> torch.Tensor:"
hindi_text = "मॉडल इनपुट टोकन को क्रमिक रूप से संसाधित करता है।"
print(len(enc.encode(english_text)))   # ~9 tokens
print(len(enc.encode(code_snippet)))   # ~16 tokens
print(len(enc.encode(hindi_text)))     # ~30+ tokens

Always measure your actual token usage. Never estimate.

Context ≠ Memory

This is the conceptual confusion that trips up most beginners and honestly, some experienced engineers too.

A context window is not memory. It’s a viewport.

Memory implies storage, persistence, the ability to recall. A context window is none of those things. It’s a fixed-size buffer that is constructed fresh for every single API call. Nothing carries over between calls unless you explicitly include it.

This has serious implications:

For agents: An agent running 50 tool calls in a loop needs to explicitly manage what goes into each LLM call. If you’re naively appending every tool result to the conversation history, you’ll hit the context limit quickly and your earlier instructions will start falling out.

For long conversations: Once a conversation exceeds your context window, you must decide what to drop. Most naive implementations just truncate from the beginning which is exactly where your system prompt and critical instructions live.

For document processing: The model does not “learn” from the document you send it. Feed it a 500-page book, it reads it once, generates a response, and forgets it entirely. Next call, you start from zero.

What Smart Engineers Do Instead

Given all of this, here’s how to actually use long context windows well:

1. Don’t fill the context — be surgical

More context is not always better. Irrelevant context adds noise. If you can retrieve and inject only the 3–5 most relevant chunks instead of 20, do it. Precision beats volume.

2. Front-load critical information

Put your most important instructions, constraints, and facts at the beginning of your context. Put the user’s question and most relevant retrieved content at the end. Leave the middle for supplementary material.

[SYSTEM PROMPT — Critical instructions]
[KEY FACTS — Things model must know]
[SUPPLEMENTARY CONTEXT — Nice to have]
[RETRIEVED CHUNKS — Most relevant at the end]
[USER QUESTION — Always last]

3. Use context compression

Before sending retrieved content to the LLM, run a compression step: use a cheap, fast model to summarize or extract only the relevant sentences from each chunk. You get the signal without the noise.

4. Implement sliding window for long conversations

Don’t truncate from the beginning. Implement a smarter strategy:

• Always keep the system prompt
• Always keep the last N messages (recency)
• Summarize older messages instead of dropping them

def compress_history(messages, keep_recent=10):
    if len(messages) <= keep_recent:
        return messages

    older = messages[:-keep_recent]
    recent = messages[-keep_recent:]

    summary = summarize_with_llm(older)  # Use cheap model

    return [{"role": "system", "content": f"Earlier conversation summary: {summary}"}] + recent

5. Measure, don’t guess

Build token counting into your pipeline from day one. Know exactly how many tokens each part of your context is consuming. Set alerts when you’re approaching limits.

def log_context_usage(system_prompt, retrieved_chunks, user_query, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)

    usage = {
        "system_prompt": len(enc.encode(system_prompt)),
        "retrieved_chunks": sum(len(enc.encode(c)) for c in retrieved_chunks),
        "user_query": len(enc.encode(user_query)),
    }
    usage["total"] = sum(usage.values())
    usage["remaining"] = 128000 - usage["total"]

    print(f"Context usage: {usage}")
    return usage

The Real Value of Large Context Windows

None of this means large context windows are useless. They’re genuinely powerful just not in the way most people use them.

The real sweet spots for large context:

• One-shot document analysis: Feed an entire contract, codebase, or report for a single analysis. No retrieval needed, no chunking errors.
• Long-form generation with reference material: Writing a technical doc while referencing a large spec.
• Complex multi-step reasoning: Where the model needs to hold many intermediate results simultaneously.
• Few-shot learning at scale: Fitting 50+ examples in context to steer behavior without fine-tuning.

In these cases, large context is genuinely useful. The key is intentionality knowing why you’re using a large context, not just defaulting to it because it’s available.

Summary: What the Context Window Actually Tells You

What you think it means What it actually means “The model can use all 128k tokens equally” Attention degrades in the middle “More context = better answers” More noise = harder retrieval “The model will remember what I told it” It’s a viewport, not memory “128k tokens = 128k words” Depends on language, format, content type “Long context replaces RAG” Different tools for different problems

Context windows are one of the most misunderstood features in modern AI engineering. The number on the spec sheet tells you the ceiling not the quality, not the reliability, not how the model actually behaves as you approach that ceiling.

Use context windows deliberately. Measure obsessively. And stop assuming that “in context” means “will be used correctly.”

This is Part 3 in my ongoing series on how LLMs really work. If you missed Part 1 (Tokens, Cost, and Prompting) and Part 2 (Sampling, Max Tokens, and Penalties), check them out this article builds directly on those foundations.

Follow me for more practical AI engineering content. I write about the gap between LLM marketing and production reality.

You Might Also Enjoy Reading!

• Write prompts like an AI Engineer
• How Google Maps Knows There Is Traffic Before You Even Reach There
• What Happens in 3 Seconds When You Pay Using UPI?
• Most Prompts Fail Because They’re Written Like Paragraphs
• Stop Building RAG Apps the Wrong Way
• Logging Is Useless — Until You Start Logging Like an Engineer
• Your Backend Doesn’t Need More If-Else — It Needs Pluggable Behavior
• Chunking in RAG Systems: Designing the Smallest Meaningful Unit of Knowledge
• RAG vs Fine-tuning: Choosing the Right Strategy for Your AI Application
• n8n Tutorial for Beginners: Build Your First No-Code Automation Workflow Step by Step
• Redux vs Zustand vs Context API: Which One Should You Use for State Management in React?
• TOON vs JSON: Choosing the Right Prompt Format for Token-Efficient LLM Systems
• Mastering React Clean Architecture: Scalable, Maintainable Frontend Engineering (Part -1)