Does Retrieval Strategy Determine Hallucination? A Controlled Comparison of Four RAG Architectures

Abstract

Retrieval Augmented Generation (RAG) has emerged as the dominant paradigm for grounding large language model outputs in external knowledge. Yet most practical deployments default to standard dense retrieval without empirically testing if more sophisticated retrieval strategies can meaningfully reduce hallucination. This post presents a controlled experiment comparing four RAG architectures — Vanilla RAG, HyDE, GraphRAG, and Agentic RAG — on 25 ArXiv AI/ML papers across 25 fact checkable evaluation questions. Results reveal that architectural complexity does not correlate with factual reliability, and that a query side augmentation technique proposed in 2022 remains the strongest approach for hallucination reduction at this corpus scale.

1. Introduction

Hallucination in large language models remains one of the most studied and least solved problems in applied NLP. RAG was introduced as a structural remedy: instead of relying on parametric knowledge baked into model weights, the model retrieves relevant documents at inference time and grounds its response in them.

The assumption underlying most RAG implementations is that retrieval quality is good enough. Embed the query, find the nearest chunks, pass them to the LLM. This works reasonably well in practice, but it raises an underexplored question: if the retrieval step fails to surface the right context, does the LLM hallucinate to compensate?

This experiment was designed to answer exactly that. By holding the LLM, corpus, and evaluation questions constant while varying only the retrieval strategy, we isolate retrieval architecture as the independent variable and measure its effect on hallucination.

2. Related Work

HyDE (Gao et al., 2022) introduced the observation that queries and documents occupy fundamentally different regions of the embedding space. A question phrased as “what dataset was used?” embeds far from a paper chunk stating “we evaluated on SQuAD 2.0” despite being semantically equivalent. Their solution was to generate a hypothetical document that would answer the query, embed that instead, and use the resulting embedding for retrieval. This approach requires no labelled data and no fine tuning.

GraphRAG (Edge et al., Microsoft Research, 2024) proposed augmenting vector retrieval with a knowledge graph constructed from the corpus. By extracting entities and relationships from documents and encoding them as graph edges, retrieval can traverse structural relationships rather than relying purely on embedding similarity.

Agentic RAG extends the ReAct framework (Yao et al., 2022) to retrieval, giving the LLM access to multiple retrieval tools and allowing it to decide iteratively which to call before committing to a final answer.

3. Methodology

3.1 Dataset

We fetched 25 recent AI/ML papers from ArXiv using the ArXiv Python API, targeting the cs.LG category. Full text was extracted from PDFs using PyMuPDF where available, falling back to abstracts otherwise. All papers were published within the same recent time window to control for topic distribution.

3.2 Evaluation Questions

25 fact checkable questions were generated from the corpus using GPT-4o-mini, one per paper. Questions were constrained to be specific and verifiable, for example:

• “What dataset was used to evaluate the proposed method?”
• “Which baseline did the authors compare against?”
• “What was the reported accuracy on the benchmark?”

Vague questions like “what is this paper about?” were explicitly excluded. Ground truth answers were extracted directly from paper text.

3.3 Architectures

All four architectures operated on the same ChromaDB vector store, the same sentence transformer embedding model (all-MiniLM-L6-v2), and the same generative model (GPT-4o-mini). Chunk size was fixed at 512 characters with 50 character overlap. Top k retrieval was set to 5 across all architectures.

Vanilla RAG embedded each query directly and retrieved the top 5 most similar chunks by cosine similarity.

HyDE first prompted GPT-4o-mini to generate a hypothetical answer to the question, embedded that hypothetical answer, and used the resulting embedding for retrieval. The original question was then answered using the retrieved real chunks.

GraphRAG used GPT-4o-mini to extract entities (datasets, methods, baselines, tasks) from each paper during indexing and stored them as a NetworkX directed graph. At query time, entities were extracted from the question, matched to graph nodes, and paper nodes reachable through graph traversal were combined with vector retrieved chunks.

Agentic RAG implemented a ReAct loop giving GPT-4o-mini access to three tools: semantic vector search, paper fetch by ArXiv ID, and keyword search over metadata. The agent issued up to five tool calls per question before generating a final answer.

3.4 Evaluation

We used a dual evaluation pipeline to avoid over reliance on any single metric.

RAGAS metrics (automated, 0 to 1):

• Faithfulness: proportion of answer claims supported by retrieved context
• Answer Relevancy: semantic similarity between answer and question
• Context Precision: proportion of retrieved chunks relevant to the question

LLM as Judge (GPT-4o, blind, 1 to 5):

• Factuality: is the answer factually consistent with the source?
• Relevance: does the answer address the question?
• Citation: are the correct papers referenced?

The judge evaluation was conducted blind — the judge model received no information about which architecture produced which answer.

Latency was recorded in milliseconds per query per architecture.

4. Results

Vanilla RAG — Faithfulness: 0.90 | Latency: 2460ms | Judge Relevance: 4.44/5

HyDE RAG — Faithfulness: 0.98 | Latency: 4142ms | Judge Relevance: 4.40/5

Graph RAG — Faithfulness: 0.93 | Latency: 3129ms | Judge Relevance: 4.40/5

Agentic RAG — Faithfulness: 0.46 | Latency: 10528ms | Judge Relevance: 2.56/5

5. Discussion

5.1 HyDE validates the embedding space hypothesis

HyDE achieved the highest faithfulness score (0.980) and the highest RAGAS scores across all three automated metrics. This directly validates Gao et al.’s core claim: the semantic gap between query space and document space is a real retrieval bottleneck, and closing it at query time produces measurably fewer hallucinations.

The cost is modest. HyDE requires one additional LLM call per query to generate the hypothetical answer. This added approximately 1682ms of latency compared to Vanilla RAG, a 68% increase in response time for a 9% improvement in faithfulness. For most production applications that prioritise accuracy, this is a favorable trade off.

5.2 Vanilla RAG remains a strong baseline

Despite being the simplest architecture, Vanilla RAG achieved the highest judge rated relevance (4.44/5) and citation accuracy (4.12/5), and was the fastest by a significant margin (2460ms). This finding is consistent with a pattern observed across machine learning research: well implemented baselines are harder to displace than their simplicity implies.

The distinction between faithfulness and relevance is important here. Vanilla RAG’s answers were judged more directly responsive to the question, while Graph RAG and HyDE sometimes retrieved broader context that was faithfully reproduced but less targeted. Being grounded and being useful are related but separable properties.

5.3 Graph RAG excels at context precision

GraphRAG achieved the highest context precision among the three non-agentic architectures (0.804), suggesting that entity graph traversal does surface more structurally relevant chunks. Its citation scores were slightly lower than Vanilla and HyDE, likely because graph traversal sometimes pulls in related papers that are adjacent rather than directly responsive to the question.

GraphRAG’s advantages are likely to compound at larger corpus scales, where the structural relationships between papers become increasingly valuable and embedding similarity alone becomes insufficient to navigate a large heterogeneous document space.

5.4 Agentic RAG: the most counterintuitive finding

Agentic RAG was the most architecturally sophisticated approach and the worst performer across every metric except none. Faithfulness collapsed to 0.460, judge factuality scored 1.96 out of 5, and latency reached 10,528ms — more than four times slower than Vanilla RAG.

The faithfulness heatmap makes this failure mode visible. Agentic RAG scored 0.00 on 11 of 25 questions, suggesting systematic rather than random hallucination. Our hypothesis is that the multi step reasoning loop amplifies rather than reduces error: each tool call introduces an opportunity for the LLM to incorporate irrelevant context, and the final synthesis step must reconcile potentially contradictory observations from multiple retrieval passes.

This finding raises a question the current experiment cannot answer: at what corpus scale does agentic retrieval begin to outperform static strategies? With only 25 papers, the retrieval space is small enough that a single well chosen vector search almost always finds the relevant chunk. The agent’s additional tool calls add noise without adding information. The value proposition of agentic retrieval likely depends on corpus size, query complexity, and the heterogeneity of the document collection.

5.5 The RAGAS and judge disagreement

A methodologically interesting observation is that RAGAS and the LLM judge did not always agree on rankings. RAGAS preferred HyDE; the judge preferred Vanilla RAG on relevance and citation. This disagreement is not noise — it reflects the genuine difference between grounding quality (what RAGAS measures) and answer utility (what the judge measures). Using both metrics exposes trade offs that either metric alone would obscure.

6. Limitations

This experiment was conducted on a small corpus (25 papers) with a single domain (AI/ML research). Results may not generalise to larger corpora, heterogeneous document types, or different question styles. The evaluation questions were LLM generated, which introduces potential bias toward question types the LLM finds easy to answer. Agentic RAG performance in particular may improve substantially at larger scales where static retrieval strategies provably fail.

7. Conclusion

Among the four architectures evaluated, HyDE offers the best trade off between hallucination reduction and added latency at this corpus scale. Vanilla RAG remains a competitive baseline that is difficult to displace on human rated quality metrics. GraphRAG shows promise for relationship heavy queries but does not clearly outperform simpler approaches on this dataset. Agentic RAG, despite its theoretical appeal, introduced more hallucination than it prevented.

The core finding is simple but worth stating plainly: retrieval strategy measurably affects hallucination, and more sophisticated strategies do not automatically produce more faithful outputs. Architectural complexity must be justified by empirical improvement, not assumed.

References

Gao, L., Ma, X., Lin, J., & Callan, J. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv:2212.10496

Edge, D., et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research. arXiv:2404.16130

Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629

Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401

Full experiment code and interactive results dashboard available at: [https://github.com/akshita270/RAG-comparison-experiment] | [https://rag-comparison-experiment.streamlit.app/]