Photo from AI

RAG Is Dead. Here Are 3 Ways It Fails in Production (And What to Build Instead)

Air Canada found out the hard way. Here’s why naive RAG hallucinates even when the right document is sitting right there.

A grieving passenger asked Air Canada’s chatbot about bereavement fares.

The chatbot told him he could book a full-price ticket and apply for the discount within 90 days. He flew. He applied. Air Canada said no. Their actual policy requires the discount before booking.

The chatbot had confidently quoted a policy that no longer existed.

Air Canada argued in court that their chatbot was “a separate legal entity responsible for its own actions.” The tribunal called this “a remarkable submission.” They lost. They paid.

This wasn’t a case of an AI model going rogue. The retrieval system had pulled a chunk from an outdated policy document. The model read it, trusted it, and answered confidently. The right kind of failure: invisible until it costs you.

This is naive RAG in production.

Welcome to Part 4 of the ‘Dead Series’. In this series, we are discussing about the issues with hyped AI technologies

Let’s understand RAG before starting.

If you want more such information about AI, consider subscribing to my newsletter, where you will get noise-free AI information every week

Link for the newsletter: Newsletter

A Visual Explanation

Photo from AI

What is RAG, and why does it exist?

If you haven’t built a RAG system before, this matters. Skip ahead if you have.

Every large language model has a knowledge cutoff. It learned from data up to a certain date and nothing after. Ask it about your internal company docs, your latest product pricing, or a policy that changed last month. It has no idea.

RAG (Retrieval-Augmented Generation) was invented to fix this.

Instead of relying on what the model memorized during training, RAG searches a knowledge base at the moment of each question, retrieves relevant documents, and hands them to the model as context before it answers.

It works like this:

User question
      ↓
[Search knowledge base]
      ↓
[Retrieve relevant chunks]
      ↓
[Insert into model context]
      ↓
Model generates answer from retrieved context

It’s a good idea. The tutorial version makes it look easy. The problem shows up when you move past the demo.

Naive RAG: what everyone builds first

The standard beginner approach:

1. Take your documents (PDFs, text files, Notion pages)
2. Split them into chunks by character count (say, every 1,000 characters)
3. Convert each chunk into a vector (a list of numbers representing meaning)
4. Store in a vector database
5. When a user asks something, convert their question into a vector too
6. Find the closest matching chunks by measuring vector similarity
7. Paste those chunks into the model’s context and get an answer

This works in demos. It stops working reliably in production.

In 2024, this was enough to ship. In 2026, most teams who’ve run it in production call it a starting point, not a solution.

The model is almost never the problem, but the retrieval is.

Photo from AI

Failure 1: The chunk problem

Air Canada’s chatbot failed partly because of how the bereavement policy was chunked.

A policy document cut by character count might split like this:

Chunk 1: "...please contact us by phone to request a"
Chunk 2: "bereavement fare. We will ask you to provide..."
Chunk 3: "...within 90 days of your travel date, send an"

Each chunk has half a sentence. No single chunk contains the complete policy. The model retrieves two fragments, stitches them together, and produces something that sounds right but isn’t.

The rule nobody tells you: chunk size is not a constant.

• Legal documents want chunks that follow section boundaries
• FAQs want chunks that pair a question with its full answer
• Technical docs want chunks at the function or procedure level

Splitting by character count treats a bereavement policy and a flight schedule the same way. They’re not.

Failure 2: Retrieval misses what matters

Vector similarity finds documents that are semantically close to the question. That’s not the same as finding the right answer.

I’ve watched teams spend weeks convinced their model was broken, when the actual issue was one of these:

• Stale data. Your knowledge base has the January policy. It’s October. Nobody updated the index after the policy changed. Retrieval pulls January’s version with full confidence.
• Embedding drift. You indexed documents in January using one model version. By September the embedding API has quietly updated. Documents from different months now live in subtly different geometric spaces. Cosine similarity between them stops being a reliable measure of semantic closeness.
• The right document, wrong chunk. The answer exists in your knowledge base. The retriever pulls three chunks that circle around it without containing it. The model fills in the gap from there.

RAG is not a hallucination solution. It’s a context injection system. The model can only be as accurate as what you put in front of it.

Failure 3: Synthesis hallucination

This one trips people up the most, and it’s the hardest to debug.

Imagine your model retrieves three factually correct chunks:

• Chunk A: “Total revenue for FY2024 was $4.2 billion.”
• Chunk B: “Year-over-year growth came in at 12%.”
• Chunk C: “The company completed the acquisition of XYZ Corp for $500 million.”

Each chunk is accurate. But the model synthesizes them and says: “Adjusted for the acquisition, organic revenue growth was approximately 8%.”

Nobody said that. The model inferred it from the gap between three correct chunks and stated it as fact.

The retrieval worked fine.

The right documents came back. Nothing in the logs flagged a problem.

That’s what makes synthesis hallucination so hard to catch in naive RAG: from the outside, it looks like a normal successful response.

Air Canada’s chatbot didn’t say it was uncertain about bereavement fares. It confidently hallucinated specific policy details because it failed to properly ground its response. The detection tools missed it.

What actually works

Naive RAG is dead. The concept isn’t.

The teams that are shipping reliable RAG in 2026 aren’t using smarter models. They’re using better pipelines.

Smarter chunking: Split on meaning, not character count. Follow document headings, section breaks, and natural paragraph boundaries. For FAQs, keep the question and answer in the same chunk. For legal docs, follow clause boundaries. Many teams now store small child chunks for retrieval but return the larger parent section in context. This gets precision on retrieval and completeness on generation.

Hybrid search: Vector similarity alone misses exact matches. Keyword search (BM25) alone misses semantic variations. Hybrid search combining keyword and semantic retrieval improves grounding accuracy by roughly 20% over either method alone. Run both, then merge results.

Reranking: After retrieval, run the top results through a cross-encoder reranker, a model specifically trained to judge whether a document actually answers the question. The top result from vector search is frequently not the most relevant result once reranked. This is one of the highest return-on-investment upgrades in any RAG pipeline.

GraphRAG for multi-hop questions: Some questions can’t be answered with a single chunk. “Which policies apply to employees who transferred between departments last year?” requires connecting entities across multiple documents.

Microsoft Research’s GraphRAG extracts a knowledge graph from your documents: entities, relationships, community summaries. It retrieves subgraphs rather than isolated chunks. In their evaluation, GraphRAG outperformed naive RAG on comprehensiveness and diversity with a 70–80% win rate.

It costs more to index.

It earns that cost on questions that require connecting dots.

Evaluation pipelines: The missing piece in most RAG deployments. You need to measure three things:

• Did retrieval find the right document?
• Did the model’s answer stay faithful to what was retrieved?
• Is the answer still accurate as your data changes over time?

Without evals, you’re shipping blind.

When to use what

Photo from AI

Key takeaways

• RAG solves a real problem. The model’s knowledge is frozen. RAG gives it live access to your data.
• Naive RAG (split by character count, cosine similarity, done) breaks in production through bad chunking, retrieval misses, and synthesis hallucination.
• The failure usually happens before the model sees a single token. Fix retrieval before touching the model.
• Hybrid search, reranking, and smarter chunking are not optional extras. They’re what turns a prototype into a product.
• You need an evaluation layer. Not after something breaks. Before you ship.

The actual lesson from Air Canada

Air Canada didn’t have a model problem. They had a retrieval problem, a staleness problem, and no evaluation layer to catch either.

The tribunal’s finding was precise: Air Canada “did not take reasonable care to ensure its chatbot was accurate.”

That’s it. That’s the whole engineering brief. Not a better model. A system that could verify what it was saying before it said it.

Most RAG failures happen before the model sees a single token. That’s where to look first.

References

• Seven Failure Points When Engineering a RAG System (IEEE/ACM, 2024) https://doi.org/10.1145/3644815.3644945
• GraphRAG: A New Tool for Complex Data Discovery (Microsoft Research, 2024) https://www.microsoft.com/en-us/research/blog/graphrag-new-tool-for-complex-data-discovery-now-on-github/
• Moffatt v. Air Canada — Civil Resolution Tribunal (2024) https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416
• RAG Is Not Enough: When Retrieval Fails in Production (Towards AI, 2026) https://pub.towardsai.net/rag-is-not-enough-when-retrieval-augmented-generation-fails-in-production-9dd2a7aa92c1
• Modern RAG in 2026: The Components That Actually Matter (Medium, 2026) https://medium.com/data-science-collective/modern-rag-in-2026-the-components-that-actually-matter-3f6a138ef117
• Real-Time Evaluation Models for RAG: Who Detects Hallucinations Best? (arXiv, 2025) https://arxiv.org/abs/2503.21157