Why RAG Goes Off the Rails Even When Retrieval Works: A Training-Format Diagnosis

Retrieval-augmented generation has become standard infrastructure for AI applications that need access to specific information beyond a model’s training data. The standard analysis of when RAG fails focuses on retrieval quality. If the right document gets retrieved, the model uses it correctly. If the wrong document gets retrieved, the model can’t use it correctly. The recommended fixes are better embeddings, hybrid search, reranking, query expansion. Each improves retrieval recall by some increment, and the field has been refining these techniques for several years.

But the standard analysis misses a deeper problem. Even when retrieval works perfectly — when the relevant document is in the model’s context — the model often fails to use the information correctly. It generates tangential responses. It mixes the document’s style into answers that should be in a different style. It drifts toward content the document mentioned even when the user asked about something else. These failures persist regardless of how good retrieval is, because they’re not caused by retrieval. They’re caused by something more fundamental about how language models process context.

This piece argues that the failure is structural to how language models are trained, that it’s predictable from the autoregressive nature of generation, and that there’s a relatively simple training modification that would fix it. The fix doesn’t require new architectures, doesn’t require massive new datasets, and doesn’t require changes to how RAG systems are designed at the application level. It requires training the underlying models to handle a context format they currently don’t handle correctly.

The argument has three parts. First, why the failure happens. Second, what the principle for fixing it should be. Third, a specific training mechanism that implements the principle.

Why autoregressive generation fails on injected content

The motivating observation for this analysis is the behavior of products that handle personalization through injection. Gemini in particular exhibits a pattern that’s hard to explain through retrieval quality but easy to explain through injection mechanics. The model frequently mentions stored facts about the user even when those facts are tangential to the current conversation. You ask it about something unrelated, and a few sentences in, your profession or location or something else from your stored profile appears in the response. The fact wasn’t relevant. It surfaced anyway.

This pattern is consistent with mid-stream injection. Some component of the system surfaces stored user facts and injects them into the context near the generation point. The model’s autoregressive process then does what it was trained to do — continues from what just preceded — and produces output that incorporates the injected content. The result is responses that mention your background tangentially because the background got injected mid-stream, and the model couldn’t help generating from it.

Front-loaded user information would behave differently. If background facts are placed at the start of context in a system prompt, the model can draw on them when relevant without being pulled toward them constantly. Selective use emerges from attention being context-dependent during generation. Facts at the start get used when the conversation makes them relevant, ignored when it doesn’t. What’s observed in Gemini’s behavior — facts surfacing tangentially even when not relevant — suggests the facts aren’t at the start. They’re being injected when the system thinks they might be relevant, and once injected, the model treats them as continuation material whether they’re actually relevant or not.

The diagnosis that follows from this observation is a structural argument about how language models are trained. Language models are trained on next-token prediction over coherent text. The training data is documents, articles, books, code, conversations — text written by humans for humans, with the property that each section follows from what came before. The style is consistent within any given piece. The topic flows. The voice is the same. Even when topics shift, they shift through transitions that the surrounding text supports.

The training objective rewards continuations that match these properties. The model is penalized for generating content that breaks the flow of what preceded it. Over billions of tokens of training, the model learns to maintain stylistic and topical continuity as a fundamental property of its outputs.

This isn’t an explicit objective specified anywhere. It emerges from the training data being naturally coherent text. The model has internalized “stay in style” as one of the things it does, because that’s what produces low loss on coherent text. The capability we call language modeling is substantially this — the ability to continue text in a way that fits what came before.

Now consider what happens when a RAG system injects a retrieved document into the model’s context. The user has asked a question in some style. Maybe casual conversational. The retrieved document is in some other style. Maybe formal academic prose, or technical documentation, or a Wikipedia introduction. The two styles get concatenated in the context window, with the user’s question and the retrieved content mashed together.

The model now has to generate an answer. Its autoregressive process is examining the context to predict the next token, and the context contains a stylistic discontinuity. The model’s training has taught it to maintain continuity, but there’s no consistent style to continue. There are at least two styles in the context, and they don’t match.

This produces the failures observed in practice. The model’s continuation tries to match the style of what just preceded. If the retrieved document just preceded, the continuation matches the document’s style — even though the question was casual. If the question just preceded but the document is the dominant content by length, the continuation might mix styles. If the boundary between them was recent, the continuation might oscillate.

The autoregressive perspective makes this concrete. At each generation step, the model computes a probability distribution over next tokens based on the hidden state, which has been built up from processing all previous tokens including the injected content. The hidden state has been shaped by the injection. When the model generates from this state, the injection’s influence is everywhere in what it produces.

Worse, the boundary tokens themselves are problematic. When the autoregressive process arrives at the first token of the injection, that token wasn’t what the model would have predicted from the preceding context. It’s a surprise. The model has to integrate the surprise while continuing to generate. Its hidden state at and after the surprise position is noisy, contaminated by the work of accommodating content that didn’t fit. The noise propagates forward through subsequent positions because every hidden state is computed from the previous ones.

By the time the model needs to produce its actual response, several things have gone wrong. Its style tracking is confused. Its hidden state representations near the injection are noisy. Its attention has been pulled toward unusual configurations to make sense of the injection’s relationship to the rest of the context. The model’s training instinct to maintain coherence is fighting against the structure of what’s actually in its context.

This is why even perfect retrieval doesn’t fix RAG. The model’s behavior when handling injected content is shaped by training that didn’t include this format. The model is doing exactly what it learned to do, which is wrong for the deployment context.

The principle: remember without generating from

The fix has to address the mismatch between training and deployment. The model needs to learn that injected content has a different role than flow content. Flow content is text the model is participating in — what the user said, what the system instructed, what the model itself has generated. The model continues flow content in the same style. Injected content is reference material — information the model should use when generating but should not continue from.

Stated as a principle: the model needs to learn to remember information from injected content without generating from it. The injection should update what the model knows, available for use through attention, without disrupting the autoregressive flow that produces the actual output.

This is a learned behavior, not an architectural feature. Standard transformers don’t have a built-in distinction between flow context and reference context. All context is processed the same way. Whether content shapes generation or just gets used as reference depends on what the model has been trained to do. Without training that includes this distinction, the model treats everything as flow.

The principle isn’t unique to RAG. The same issue arises whenever a system injects content into the model’s context that wasn’t part of the conversation flow. Memory systems that surface relevant past information. Personalization systems that pull in user context. Tool use where the output of a tool gets concatenated into the context for the model to use. Each of these is an injection, and each is subject to the same autoregressive disruption.

Products that handle these features well share a structural property: they front-load information at the start of context rather than injecting it mid-stream. Information at the start gets integrated into the hidden state from the beginning, becomes part of the foundation the rest of generation builds on. There’s no discontinuity to recover from because there’s nothing yet to be discontinuous with. Models handle this format reasonably well even without specialized training because it doesn’t violate the assumption that context is coherent.

But many useful product features require mid-stream injection. You can’t always know what information the model will need before the conversation starts. Real systems need to inject content based on what’s happening in the conversation, which means injection at the time the relevance becomes clear. The choice between front-loading and mid-injection is really a choice between limiting what your product can do and accepting the disruption that comes with mid-injection.

Before going further, it’s worth being explicit about when mid-injection is actually necessary versus when it’s a habit that could be replaced with front-loading. Context windows are constructed programmatically. A program builds the context for each turn before sending it to the model. The model processes the entire context fresh on every forward pass; there is no persistent state that depends on where content appeared in previous turns. So the program can put tokens anywhere it wants in the context window, regardless of when the underlying decisions were made.

This means a substantial fraction of cases that look like mid-injection don’t actually need to be. If you know what to retrieve before generation starts — based on the user’s current question, before the model produces any output — you can do the retrieval, then construct the context with the retrieved content at the front, then run the model. The retrieval happened mid-conversation in the sense that it occurred between turns, but the resulting context has the retrieved content at position zero, with the conversation history afterward. The model sees front-loaded content, not mid-stream injection.

For cases that work this way, front-loading is the right answer and no special training is needed. Put the retrieved document at the start of the context with a brief instruction identifying what it is. The model handles this format reasonably well because it doesn’t violate the autoregressive assumption that context is coherent. The instruction provides enough framing for the model to use the document as reference rather than continuation.

The cases where front-loading doesn’t work are the ones where you genuinely can’t know what to inject until generation is already underway. Tool use, where the model decides mid-response to call a function and the function’s output has to come back into context. Retrieval triggered by what the model is currently generating, where the query for retrieval is itself produced by the model and the result has to be incorporated. Memory systems that surface relevant past information based on the model’s current context. These are cases where the injection point is forced by the timing — you can’t put the content at the start because you didn’t know to retrieve it until the model was already producing output.

For these cases, the no-op training is the actual fix. For everything else, just front-load. The principle says: don’t accept the disruption when you don’t have to. Move what you can to the start of context. Train the model to handle injection correctly only for the cases where injection is unavoidable.

The mechanism: make no-op the label

We make no-op the label.

That single sentence specifies the entire mechanism, and stating it that way tells you exactly how the learning works. During training, injected content is marked with opening and closing meta tags. At every position from the opening tag through the closing tag, the label is a special no-op token. Standard next-token prediction with standard gradient descent does the rest.

The model learns to predict no-op at those positions the same way it learns to predict any other token: through gradient descent on the standard cross-entropy loss between predictions and labels. There is no special handling of the loss function. There is no exclusion of positions from the gradient. The mechanism is just standard training with one specific choice of label at the relevant positions.

What the model ends up learning is a simple rule. Between an opening meta tag and a closing meta tag, predict no-op. Outside those tags, predict the next token in the document. Both behaviors come from the same training objective. The model isn’t doing anything architecturally unusual. It’s learning to produce a particular token in a particular context, the way it learns to produce particular tokens in particular contexts for everything else.

At inference, the model generates no-op tokens when it encounters meta tags in its input. The generation process produces them just like any other token. The decoder, knowing that no-op tokens shouldn’t be displayed, omits them from what gets shown to the user. The model went through the motion of generating during the injection span, but the user never sees those generations. Once the model exits the closing tag, it generates normal tokens again, and those get displayed.

The attention mechanism is unrestricted throughout. The model attends to the injected content from positions both inside and outside the tags. The information becomes part of the model’s representations. When the model is generating outside the tags and producing real output, it can use the injected information through attention. But the model isn’t generating from the injection in the sense of continuing its style, because during the injection span the model’s prediction was no-op, not the injection’s actual tokens. The model never produced flow content based on the injection’s style because the training signal taught it not to.

This produces the behavior the principle requires. The model attends without generating from. The injection becomes available information without becoming the basis for continuation. The autoregressive flow that produces visible output resumes cleanly after the closing tag.

Training data construction

The training data has to satisfy a specific property for this to work. The content outside the meta tags has to actually use information from the content inside the meta tags. Otherwise the model has no signal teaching it that the injected content is useful. It would learn to predict no-op during injection and then continue without reference to what was inside the tags, which is exactly the failure mode we’re trying to avoid in a different form.

The good news is that the right training data already exists in the world. Technical papers cite other papers. The citing paper’s text uses information from the cited paper. The cited paper’s content is exactly the kind of injected reference that the citing paper’s continuation draws on. To construct training examples, you take a paper and its cited references. For each citation in the paper, you find the cited paper. You insert the cited paper into the source paper’s context, wrapped in meta tags, at a location before the citation appears. The source paper’s original text continues from that point, and the continuation already references the cited material in the way RAG would want a model to reference an injected document.

No synthesis is required. The relationships are real. The continuations are authored, not generated. The source paper’s writer was doing exactly what we want the model to learn — using the cited information to inform their continuation while writing in their own voice.

You can scale this up across the academic literature. Hundreds of thousands of papers, each citing dozens of others, produce millions of training examples. The construction is mechanical. Identify citations, fetch the cited papers, insert them with meta tags, label the meta-tagged positions as no-op. The rest of the source paper provides the labels for normal next-token prediction.

The same approach works for non-academic content with citation-like structures. News articles that quote sources. Books with footnotes. Wikipedia articles with references. Technical documentation that references manuals. Any content where one document explicitly draws on another can become training data for handling injection correctly.

The architecture doesn’t need changes. Transformers already handle special tokens. Adding a no-op token to the vocabulary is trivial, and the meta tags are just additional special tokens. The training pipeline doesn’t need modifications to its loss function — the no-op token is treated like any other token during training, with gradient flowing normally through standard cross-entropy loss. The only modification is at inference: the decoder filters no-op tokens from the displayed output. That’s a small change in the generation loop.

The training cost would be modest compared to full pretraining. This could be a fine-tuning stage on top of an existing model, exposing the model to the injection format and the no-op behavior on top of capabilities it already has. The resulting model would handle mid-context injection cleanly while retaining its existing language modeling capability.

What this fixes and what it doesn’t

The proposed mechanism fixes the autoregressive disruption caused by mid-context injection. A model trained with the no-op label inside meta tags would maintain conversational style across injections, use information from injected content when relevant, and avoid the tangents and style drift that current models exhibit when handling RAG-style context.

This is a substantial fix because it addresses a problem that’s not currently being addressed at the right level. Most attempts to improve RAG focus on retrieval quality, prompting strategies, or architectural changes to how documents get incorporated. The training-level fix is more fundamental. It changes what the model has learned to do, not what we ask it to do at inference.

The mechanism doesn’t fix retrieval quality directly. If the retrieval system fetches the wrong document, no training modification will help the model use the wrong document correctly. The diagnosis here is that retrieval-quality fixes don’t fully solve RAG because there’s a downstream problem in how injected content gets used. Both layers need attention. Better retrieval plus better injection-handling produces a system where each layer is doing its job correctly.

The mechanism also doesn’t fix the lost-in-the-middle problem entirely. That problem has multiple causes, of which autoregressive disruption is one. Other contributors include the position bias of attention mechanisms, the way long contexts get summarized in early layers, and the training distribution’s lack of long-range dependencies that require detail extraction from middle positions. The no-op label approach addresses the disruption component without addressing these other components.

What the fix does is remove a specific failure mode that’s currently bundled into the broader category of “models don’t use context reliably.” Distinguishing this failure mode from others is itself useful. It points at a specific intervention with a clear mechanism. Other failure modes need other interventions, but at least one of them is now identified and addressable.

Why this hasn’t been done

The mechanism is simple and the implementation is tractable. The reason it hasn’t been done at scale isn’t technical difficulty. It’s a combination of factors related to how the field is organized.

Most foundational model training is done by a small number of large labs. Their training pipelines are optimized for the training objectives they’ve committed to, which are mostly variants of next-token prediction on whatever data they have. Adding new training objectives requires either modifying the pipeline or running additional training stages, both of which are expensive and require justification within the lab’s research priorities.

The justification for this specific fix would require recognizing the failure mode it addresses, which isn’t widely articulated in the form proposed here. Most analysis of RAG failures attributes them to retrieval quality. Without the diagnosis pointing at autoregressive disruption specifically, there’s no clear motivation for the training-level fix. This is a problem that’s hidden by how it’s usually framed.

There’s also an implicit assumption in the field that better instruction-following can be achieved through reinforcement learning from human feedback. RLHF can compensate for some autoregressive disruption by training the model to produce responses that humans rate well. But RLHF works on the level of overall responses, not on the level of individual generation steps near injection boundaries. It’s downstream of the disruption, treating the symptom rather than the cause. The training-level fix proposed here addresses the cause, which RLHF then doesn’t have to compensate for.

Finally, there’s the practical issue that fixing this requires generating training data in a format that doesn’t naturally exist. Coherent text doesn’t come pre-marked with meta tags around the parts that should be reference material. Generating the training data is work, even if the work is straightforward. For labs whose teams are already at capacity working on other priorities, adding this work requires deciding it’s important enough to displace something else.

What’s untested

Everything in this piece has been structural argument about how transformers work and what training would change. None of it has been empirically tested in the form proposed here. Several questions that the piece treats as resolved are actually open, and the answers might cut against parts of the analysis.

The most basic empirical question is whether front-loading actually produces better results than mid-injection in practice. The autoregressive disruption argument says mid-injection causes problems through style discontinuity. But there’s a competing argument from locality. Attention in transformers tends to be stronger to recent tokens. If retrieved content is placed right before the position where the model needs to use it, attention has the shortest path. If it’s at the start of a long context with thousands of tokens of conversation in between, the model’s attention to it at generation time might be effectively zero. Lost-in-the-middle is a real phenomenon. Front-loaded content might be forgotten in practice even though it’s structurally cleaner.

So the comparison isn’t simply front-loading versus mid-injection. It’s a tradeoff between two failure modes. Mid-injection produces autoregressive disruption. Front-loading risks insufficient attention at generation time. Which dominates probably depends on context length, how the retrieved content relates to the query, how the model was trained, and possibly other factors not yet identified. The right answer in different conditions might be different. The piece’s recommendation to front-load when possible is plausible but not proven.

The empirical experiment that would settle this is straightforward in shape but hasn’t been run in the form needed. Take a model. Take a corpus of queries that benefit from retrieved context. Run each query in two conditions: front-loaded retrieval at position zero, and mid-stream injection at the point of relevance. Compare answer quality. Repeat across context lengths and document types. The result tells you how the tradeoff shakes out empirically.

The next question is whether the no-op training actually produces the predicted behavior. The piece reasons from how transformers process tokens to predict that training with no-op labels at injection positions would cause the model to attend to injected content without generating from it. But “the model would learn to attend without generating from” is a prediction, not an observation. The training might fail in several ways. The model might learn to ignore tagged content entirely rather than using it through attention. The training might be unstable. The behavior might generalize badly from training to inference. The model might overfit to the specific format of training examples and fail on slightly different formats. Without running the experiment, none of this is settled.

The experiment for this part would require fine-tuning a model with the no-op label format on appropriate training data, then evaluating whether the resulting model handles injected content correctly. Concrete metrics would include: does the model produce relevant continuations after closing tags, does it use information from inside tags when generating outside them, does it maintain conversational style across injection boundaries, does it perform better than the same base model without no-op training on the same evaluation tasks. Each of these is measurable but none has been measured.

A third question is whether the locality argument matters more than the disruption argument for current models. Models trained with substantial RLHF have already absorbed some compensation for injection-format inputs. The base autoregressive instinct is real but it’s been partially overlaid with instruction-following capabilities that handle some retrieval formats reasonably well. The marginal benefit of no-op training on top of existing RLHF might be smaller than the analysis suggests. Or it might be larger because RLHF only papers over a base capability mismatch that the no-op training would address directly. Without empirical comparison, the magnitude of the benefit is unknown.

The piece’s argument is structural rather than empirical. The structural argument is that the training-deployment mismatch produces the failure modes observed in practice, that the principle of “remember without generating from” addresses the mismatch, and that the no-op label mechanism is one way to implement the principle. Each of these claims is reasoned from how transformers work. But the field has been wrong before about predictions reasoned from architectural principles. The experiments matter.

The honest framing is that this piece proposes a hypothesis with structural support and concrete experimental tests. The hypothesis: mid-context injection produces autoregressive disruption that’s distinct from retrieval quality and could be addressed by training. The proposed test: train a model with no-op labels at meta-tagged positions, evaluate whether injection handling improves. The proposed comparisons: front-loaded versus mid-injection performance across context lengths, with and without no-op training. The result would either support the structural argument or reveal what’s missing from it.

If the experiments support the analysis, the no-op training becomes a real engineering proposal that frontier labs could adopt. If the experiments contradict the analysis, the failure mode is informative — it would tell us something about how transformers actually handle context that current architectural reasoning is missing. Either outcome advances the question.

This piece is a starting point, not a finished result. The thinking is structural and the predictions are concrete enough to be tested. Whether anyone with the resources to test them will do so is uncertain. But at least the questions are now in a form where they could be addressed empirically rather than remaining as informal observations about why RAG doesn’t work as well as it should.

Implications for product design

Even without the training-level fix, the diagnosis has implications for how RAG and personalization features should be designed.

Front-load context whenever possible. Information available at the start of conversation should go in the system prompt or at the beginning of the user’s first message. The model handles this format reasonably well even without specialized training because it doesn’t violate the autoregressive assumptions.

Avoid mid-stream injection unless necessary. The current generation of models will exhibit the failure modes described here when content is injected mid-conversation. Until the underlying models are trained to handle this format, products that rely heavily on mid-injection will produce inconsistent outputs. Memory systems that pull in past information should consider whether the information could be presented at the start of the session rather than injected when triggered.

When mid-injection is necessary, format it consistently. If your product injects content, do it the same way every time. The model has some chance of learning to handle a consistent format through whatever fine-tuning has occurred, even if not through the specific mechanism proposed here. Inconsistent injection formats make the model’s job harder.

For product teams that have model training capability, consider doing the training proposed here. The cost is modest and the result is a model that handles your product’s actual context format correctly. Generic foundation models won’t do this for free; they need to be trained for the format they’ll be deployed on.

Closing

The diagnosis here is that RAG and similar features fail in a specific way that isn’t captured by the standard “retrieval quality” framing. The mechanism is autoregressive disruption: models trained on coherent text behave incorrectly when their context contains discontinuities, because their training has taught them to maintain continuity that injection violates.

The principle for fixing this is that models need to learn to absorb information from injected content without generating from it. The injection should update what the model knows, available through attention, without disrupting the autoregressive flow that produces the output.

The mechanism for implementing this principle is meta tags with no-op labels. Mark injected content with special tokens. During training, label every position inside the marked span as no-op. The model learns to predict no-op for those positions. Outside the tags, normal next-token prediction continues. The model attends to the injected content from any position but generates from it nowhere, because the labels at injection positions were no-op rather than the injection’s actual tokens.

The training data can be constructed without synthesis. Technical papers and their citations provide the right structure ready-made. The citing paper’s content already uses information from the cited paper while remaining in the citing paper’s voice. Wrap the cited paper in meta tags and insert it before the citation point in the source paper. The source paper’s continuation is the training target for normal next-token prediction. The cited paper, inside the tags, gets no-op labels. Millions of such examples exist across the academic literature alone. Similar structures exist in news, books, technical documentation, and any content with explicit reference relationships.

This proposal doesn’t require new architectures. It doesn’t require massive new datasets in a different sense — the data already exists, just in a form that needs to be reformatted. It doesn’t require expensive training from scratch. It requires a fine-tuning stage on the reformatted data with the right loss labels. The work is tractable. The result would be a model that handles mid-context injection cleanly, fixing one of the more frustrating failure modes of current AI products.

Whether the specific mechanism here is the best implementation of the principle, I don’t know. There are other ways to instantiate the same principle. Architectural changes that separate flow processing from reference processing. Attention masking schemes that distinguish different roles for different parts of context. Different training formats with different markers and loss structures. The mechanism proposed here is a starting point, not necessarily an ending point.

The principle, though, seems solid. Mid-context injection disrupts autoregressive generation. The model needs to learn to handle injection without disrupting the flow. Whatever mechanism accomplishes this, the result would be a substantial improvement over current behavior. The field would benefit from making this fix, in whatever form, a higher priority than it currently is.