Understanding the LLM Attention Mechanism: Why the Middle Fades

Why Does AI Ignore the Middle of Your Message?

The Research Gives an Answer That Changes How You Prompt.

The Science Behind the U-Curve

The model doesn’t read your prompt the way you read a page. Research from Stanford, MIT, and Anthropic confirms that everything you put in the middle of a long message gets attended to least. Here’s what that means for every interaction you have with AI — and the three specific changes that fix it.

Episode 4 of The Honest Machine — how AI actually works, one mechanism at a time

There is a specific feeling you get when AI misses something that was right in front of it.

Not a hallucination — the information was there. You gave it. You can see it in the chat, or in the document you pasted, or in the instructions you wrote out carefully. The AI just didn’t treat it with the weight you assumed it would.

You rewrite the message. You move the important part to the top. Suddenly it works perfectly. You probably assumed you had explained it badly the first time. You didn’t. Where you put it was the problem.

Most people discover this by accident and file it away as a quirk. It's not a quirk. It is the architecture — one of the most consistent, well-documented, and practically consequential properties of how these systems actually work.

Once you understand it, something shifts: you stop blaming your writing and start placing your instructions where the model actually attends to them.

What Attention Actually Is?

The previous three episodes of this series established three things.

The model generates text by predicting the next word — one at a time, building on what came before. It learned those predictions from an enormous archive of internet text, with all the biases and gaps that entailed. And RLHF training tuned it to optimise for human approval, which is why it agrees with you even when you’re wrong.

Episode 4 explains the fourth layer:

how the model decides, for each word it generates, which parts of your input actually matter.

When you read a page, your eyes move left to right, top to bottom. You process the words sequentially. Then you move to the next paragraph. The model does not do this.

For every single word it generates, the model looks back at the entire conversation — every message you have written, everything it has responded with — and assigns a weight to each part of it. Some parts get high weight, meaning they strongly influence what word comes next.

Other parts get lower weight, meaning they fade into the background. The model generates each word by attending to everything that came before — with different weights on different parts.

This process is called attention. It is the central innovation in the 2017 paper “Attention Is All You Need” by Vaswani and colleagues at Google, which introduced the transformer architecture that underlies every major AI model you use today.

The name sounds abstract. The implication is concrete: when you write a message to an AI, different parts of that message carry different weight depending on where you put them.

For each word the model generates, it looks back at the entire context and assigns attention weights

The U-Shaped Curve — What the Research Confirmed

In 2023, a team of researchers at Stanford and the University of North Carolina ran a series of experiments to understand how models actually use long contexts. They gave models documents containing thousands of words and buried specific pieces of information at different positions — beginning, middle, and end. Then they asked questions about that information.

The finding is now one of the most cited results in large language model research: model performance followed a U-shaped curve depending on where the relevant information was placed.

Information at the beginning of the context: high accuracy. Information at the end of the context: high accuracy. Information in the middle of the context: accuracy dropped sharply.

They called the paper “Lost in the Middle: How Language Models Use Long Contexts.” It was published in the Transactions of the Association for Computational Linguistics in 2024. It has been replicated, extended, and confirmed across multiple model families, multiple context window sizes, and multiple task types.

The U-curve holds for context windows of 4,000 tokens. It holds for 32,000 tokens. In 2025, researchers confirmed it persists in models with context windows of 128,000 tokens and beyond. Making the window bigger does not solve the problem. It makes the middle bigger — which means more space for information to get lost.

The middle is where information goes to disappear.

Why It Happens?

The Architectural Explanation:

For two years after Liu et al. published the original paper, the mechanism behind the U-curve was unclear. Researchers could see the pattern but could not explain precisely why it arose.

In 2025, a team at MIT provided the mathematical explanation.

The reason comes down to something called causal masking — a fundamental property of how transformer models process text. Models generate text left-to-right: token #1, then token #2, then token #3. At each step, a token can only “see” the tokens that came before it, not the ones that come after. This is causal masking. It is why the model generates text in order rather than all at once.

Here is the subtle consequence: token #1 is visible to every single subsequent token. Token #2 is visible to token #3, #4, #5, all the way to the end. Token #500, sitting in the middle, is only visible to tokens #501 onward.

The MIT researchers framed this using graph theory: causal masking creates an asymmetric directed structure where early tokens lie on exponentially more computational paths than middle tokens.

Think of it this way. Every time the model generates a new word, it is implicitly referencing the beginning of your message many times over — because the beginning has been “present” at every generation step. The middle of your message has only been present for some of those steps.

The end has been present for the fewest steps, but benefits from recency bias: it is the most recent signal before the model responds.

Beginning: attended to by everything that follows it.

End: the most recent context, strongest immediate signal.

Middle: neither anchor nor recent. Structurally disadvantaged.

There is something almost poignant about this. The pattern is not unique to AI. Psychologists have documented the same U-shape in human memory since Hermann Ebbinghaus first described it in 1885. We remember the first items on a list — the primacy effect. We remember the last items — the recency effect. The middle items are the hardest to recall.

The transformer architecture and the human memory system arrived at the same shape through completely different mechanisms.

Four Situations — and How to Handle Each One

This is where the episode shifts from understanding to practice. The U-curve is not an academic curiosity. It directly affects the quality of every interaction you have with AI. Here is what it means in four common situations — with before-and-after examples for each.

Situation 1: You give the model a long document to analyse

The wrong structure and the most common one:

Here is a report I need you to analyse.

[20-page report. The key finding you want the model to focus on 
is on pages 8–10 — right in the middle.]

Please summarise the main conclusions and flag any risks.

The model reads this and attends strongly to the beginning (your instruction) and to the end (your final question). The critical section on pages 8–10 is in the U-curve’s blind spot.

The right structure:

I need you to focus specifically on [the section about supply chain risks]. 
This is the most important part of what follows.

[20-page report]

Remember: the section on supply chain risks is the priority.
Summarise the main conclusions with special attention to that section.

You have placed the instruction at the beginning (high attention), flagged the important section explicitly, and repeated your core requirement at the end (recency effect). The model’s structural bias now works with you, not against you.

Situation 2: You give multiple instructions in a single message

The wrong structure:

Please review this code. Make sure to check for performance issues, 
security vulnerabilities, readability, deprecated functions, and edge cases. 
Also look at the error handling.

[500 lines of code]

Thanks, looking forward to your review.

The instructions are in the middle of the first sentence, surrounded by social framing. The 500 lines of code push them further toward the middle of the overall context. “Deprecated functions” and “edge cases” are the items most likely to be underweighted.

The right structure:

Review this code for exactly these five things:
1. Security vulnerabilities
2. Performance issues  
3. Deprecated functions
4. Edge case handling
5. Readability

[500 lines of code]

Before responding, confirm you checked all five items above.

The numbered list at the beginning is easy to attend to. The explicit “before responding” instruction at the end activates recency bias to reinforce the checklist. The code — which is data, not instruction — sits in the middle, where it belongs.

Situation 3: You are working on a long conversation

Every exchange you have adds new context. As the conversation grows, your earliest messages move further from the end — which means they drift toward the middle. The original instructions you gave in message one are now competing with everything that came after them.

This is why long AI conversations often drift. The model is not “forgetting” your original instructions. It is weighting them less, because they are no longer at the end and the subsequent conversation has diluted their signal.

There are two fixes.

First: for any task that requires many exchanges, periodically restate the core requirement. Not because the model forgot it. Because you are refreshing its position in the attention weighting.

Second: for very long tasks, consider starting a new conversation and summarising the key decisions made so far. The summary moves everything important back to the beginning of a fresh context.

Situation 4: You want the model to use a specific piece of information

Anthropic’s researchers discovered something striking when testing Claude 2.1’s long-context performance. When they simply added a sentence before the relevant content — explicitly pointing to it as the most important part — accuracy jumped from 27% to 98%.

That is not a small improvement. That is what happens when you work with the attention mechanism instead of assuming it distributes evenly.

The practical version of this finding: if there is a specific sentence, paragraph, or fact that must inform the model’s response, do not trust that the model will find it by reading your entire context. Point to it explicitly. Elevate it. Say “the most relevant part of the following is…” before it appears.

The model is not resistant. It is just attending to everything at once, and it needs signals about what matters most.

The One Practical Rule That Covers Everything

If you remember nothing else from this episode, remember this:

Your most important instruction goes first. Your most important constraint gets repeated at the end. Everything else — the context, the data, the document — goes in the middle.

This is not a workaround for a flaw. This is working with the architecture as it actually exists. The beginning and end of your context are the parts the model attends to most. Use that structure deliberately.

The Complete Map So Far

Four episodes of this series have now given you something that most AI users do not have: an accurate mental model of the system they use every day.

Episode 1 — The model generates each word by predicting what comes next, based on patterns learned during training. Fluency and accuracy are different things. The model produces both in exactly the same voice.

[embed]It’s Not Thinking. It’s Completing- How LLMs Actually Generate Text? The Honest Machine — Episode 1: Next-Token Prediction, Temperature, and Why Fluency ≠ Accuracymedium.com

Episode 2 — Those patterns were learned from an enormous archive of internet text — biased toward certain voices, certain countries, certain types of content, and frozen at a specific point in time.

[embed]LLM: The Student Who Read Everything. The Honest Machine — Episode 2: Pretraining, the Archive, and Three Structural Problems Every LLM Inheritedmedium.com

Episode 3 — RLHF training then tuned the model to optimise for human approval, which is why it agrees with you more confidently when you push back. Not because you convinced it. Because the training preferred agreeable responses.

[embed]Why AI Gets More Confident When You Push Back.It’s Not Because You’re Right Scientists Proved AI Models Will Agree With Both Sides of Any Argument. The Reason Should Change How You Use AI Every…medium.com

Episode 4 — When it generates each word of its response, it weighs the beginning and end of your input most heavily. The middle of your message is structurally disadvantaged by the architecture.

Together: a model that completes patterns from a biased archive, trained to be agreeable, that reads your messages with a U-shaped attention bias toward the beginning and end.

This is not a reason to stop using these tools. It is a reason to use them differently. The model is genuinely useful. It is also genuinely not what most people think it is.

Your prompt has a blind spot. Now you know where it is. Put the things that matter most outside of it.

In Episode 5 — “The Temperature Problem” — we will explain why the same prompt produces different answers every time, what randomness is doing inside the model, and how to control it for tasks that need precision versus tasks that need creativity.

Sources — all primary:

• Nelson F. Liu, Kevin Lin, John Hewitt et al., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics, 12:157–173, 2024. Stanford/UNC. The original “lost in the middle” finding — U-shaped attention curve across all context window sizes.
• Wu et al., “Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias,” arXiv:2603.10123, March 2026. MIT. Mathematical proof that causal masking creates an asymmetric directed graph where early tokens lie on exponentially more computational paths.
• Hsieh et al., “Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization,” MIT and Google Cloud AI, 2024. Demonstrated U-shaped positional attention bias independent of content relevance; proposed calibration mechanism.
• Xiao et al., “Efficient Streaming Language Models with Attention Sinks,” 2023. First formalization of attention sinks — early tokens absorbing disproportionate attention mass due to their position.
• Anthropic internal testing of Claude 2.1: adding “Here is the most relevant sentence in the context” improved long-context retrieval accuracy from 27% to 98%. Reported in: davidwsilva.substack.com, June 2025.
• Vaswani et al., “Attention Is All You Need,” NeurIPS 2017. The original transformer architecture paper introducing the attention mechanism.
• Ebbinghaus, H. (1885). Über das Gedächtnis (Memory: A Contribution to Experimental Psychology). The original documentation of the serial position effect — primacy and recency in human memory.