The Real Tradeoff Between GraphRAG, Vector RAG, and Hybrid RAG

In 2022, I finished my Master’s thesis on abstractive text summarization. I trained LSTM encoder-decoder models on review data, tuned learning rates by hand, and celebrated when my ROUGE scores beat the reference paper by a few points. That was NLP then: you trained a model for one task, and the model was the system.

Four years later, the question has changed. Nobody asks “Can the model summarize?” anymore. The question now is: when a language model needs knowledge it wasn’t trained on, what is the best way to hand it that knowledge?

Most articles about GraphRAG read like marketing. “Better than vector RAG! Handles multi-hop reasoning! Reduces hallucinations.” Almost none publish numbers, costs, or tradeoffs. So I ran the experiment myself: three retrieval systems, one corpus of 500 FDA drug labels, and thirty questions across six categories designed to test specific retrieval capabilities. I measured accuracy, latency, tokens, cost, and, for the graph system, how often it needed to correct its own mistakes.

Hybrid hits 62% accuracy, GraphRAG alone hits 55% & Vector RAG hits 28%. But the more interesting result isn’t accuracy; it’s cost. GraphRAG is cheaper per query than Vector RAG (1,356 tokens vs 2,905), which inverts the usual narrative. The rest of this article unpacks where each system wins, where each fails, and which you should use when.

Before going further, I want to be clear about two things. First, I am not a clinical expert. This is a technical benchmark of retrieval architectures that happens to use medical data because it is public and well-structured & nothing here is medical guidance, and none of the system outputs should inform any health decision. Second, nothing in this article is meant to challenge anyone else’s published results. This is one experiment on one corpus, with one evaluation design. Treat my conclusions as informed opinion and working theory which is a starting point for your own measurements, not a verdict on anyone else’s.

The Architecture & Setup

1. Corpus: ~500 drug labels pulled from the openFDA API, filtered to labels with a generic_name and an indications_and_usage section. The corpus skews toward OTC products because of openFDA’s default ordering. That’s a real limitation I’ll come back to.

2. Knowledge Graph: GPT-4o-mini extracts structured entities and relationships from each label in JSON mode with temperature=0. The resulting graph (Neo4j AuraDB Free) uses the following schema:

Nodes:Drug, Condition, AdverseEvent, Ingredient, Warning
Edges:(Drug)-[:TREATS]->(Condition)
(Drug)-[:CONTRAINDICATED_IN]->(Condition)
(Drug)-[:HAS_ADVERSE_EVENT]->(AdverseEvent)
(Drug)-[:CONTAINS]->(Ingredient)
(Drug)-[:INTERACTS_WITH]->(Drug)
(Drug)-[:HAS_WARNING]->(Warning)

At the end of extraction, the graph has ~500 drugs, ~750 conditions, ~820 adverse events, and ~600 warnings.

Vector RAG: Drug labels are chunked into 400-word segments with a 50-word overlap. They are embedded with all-MiniLM-L6-v2 (384-dim, free, CPU), stored in ChromaDB, and queried by embedding the question, retrieving the top-5 chunks, stuffing them into the prompt, and generating the answer with GPT-4o-mini.

GraphRAG: GPT-4o-mini generates a Cypher query from the natural-language question, runs it against Neo4j, and synthesizes a natural-language answer from the structured results. If the first Cypher attempt returns empty results or OUT_OF_SCOPE, the system retries up to twice, using the failure history as feedback.

Hybrid: Runs both retrievers in sequence. Graph facts (structured) and vector chunks (prose) are concatenated into a single context. A single LLM call generates the answer, with explicit guidance to “prefer graph for counts and lists, prefer vector for nuance.”

Evaluation: 30 hand-written questions split into 6 categories:

Each question has a hand-written gold answer. GPT-4o scores each system answer on a 0/1/2 rubric (wrong / partial / correct). Using the LLM as the judge is now standard practice, but it comes with known caveats, which I will return to in the limitations section.

The three design decisions that made GraphRAG work

1. Ground the schema in real data, not descriptions.

The Cypher generation prompt does not describe the schema in prose. It queries the live knowledge graph at runtime and embeds actual example values:

Drug(name, dosage_form, brand_names)
name examples: "naproxen", "trametinib", "glimepiride", ...
dosage_form examples: "oral", "topical", "ophthalmic", ...

The model doesn’t know whether dosage_form holds values like “oral” or sentences like “administered orally twice daily.” That uncertainty produces malformed queries and false refusals. With grounding, property lookups become trivial. And because the examples are queried live, the prompt stays synchronized with the data forever, with zero maintenance.

2. Retrieve your few-shot examples, don’t hardcode them

Instead of fixed examples in the prompt, I built a library of 32 curated (question → Cypher) pairs spanning all six categories. At query time, the user’s question is embedded, and the three most semantically similar examples are injected.

An aggregation question sees aggregation examples. A negation question sees NOT EXISTS patterns. The prompt adapts to the question. This reuses the same embedding model and the same similarity machinery as the vector RAG system. The retrieval infrastructure retrieves examples for the graph system. The two competitors share a component.

3. Self-correcting retry loop

If the generated Cypher returns empty results, an error, or OUT_OF_SCOPE, the failure is fed back to the LLM with a list of common failure modes (“match was too restrictive — try CONTAINS instead of =”). Up to 2 retries per query. This pattern generate, check, regenerate with feedback is sometimes called “self-refine”. It’s cheap, effective, and adds real robustness without changing the underlying model.

20% of all GraphRAG queries needed at least one retry to succeed. One in five. Without the loop, GraphRAG’s accuracy would drop substantially, concentrated exactly in the hardest categories. I’ll show where the retries fire below, because the distribution itself turned out to be diagnostic.

Results:

Three observations from the table:

Hybrid wins overall, by a meaningful margin. 62% beats GraphRAG’s 55%, beats Vector’s 28%. Hybrid is best or tied on 5 of 6 categories. The combined-context approach retrieves structured facts from the graph (lists, counts, relationships) and prose context from vector chunks (nuance, explanation). Where a graph or a vector fails individually, having both signals available lets the LLM recover.

Vector RAG fails where you’d expect. 0% on aggregation, 0% on negation. Vector retrieves chunks of prose; it can’t count drugs without enumerating every chunk it retrieved, and it can’t reliably reason about “things that don’t have property X.” This isn’t a flaw, it’s the wrong tool for those jobs.

GraphRAG handles structured queries cleanly. 80% on multi-hop, path, and single-hop. Where the answer is “find all entities satisfying constraints A, B, C,” Cypher is the right interface. The 40% on aggregation looks lower than it should, but it reflects issues with gold answers, which I’ll discuss in the limitations.

Examples

Example 1: where GraphRAG wins decisively

**Question: **Find drugs that treat cancer and have hemorrhage as an adverse event.

Vector RAG:

“Insufficient information in the provided excerpts.”

GraphRAG:

“The drugs that treat cancer and have hemorrhage as an adverse event are trametinib.”

This is the canonical multi-hop case. Vector retrieves 5 chunks; none of them happen to mention both “cancer” and “hemorrhage” together because they live in different sections of different drug labels. Graph traversal walks the explicit relationships and finds the intersection directly. The generated Cypher:

MATCH (d:Drug)-[:TREATS]->(c:Condition), 
(d)-[:HAS_ADVERSE_EVENT]->(e:AdverseEvent)
WHERE (toLower(c.name) CONTAINS 'cancer' 
OR toLower(c.name) CONTAINS 'tumor' 
OR toLower(c.name) CONTAINS 'melanoma') 
AND toLower(e.name) CONTAINS 'hemorrhage'
RETURN DISTINCT d.name LIMIT 50

You can’t synthesize this kind of cross-document join from vector retrieval. The information isn’t in any single chunk.

Example 2: the retry loop saving a query

**Question: **Which drugs share an active ingredient with naproxen?

Vector RAG:

“Insufficient information in the provided excerpts.”

GraphRAG, first attempt: Generated Cypher with overly-strict matching (toLower(d.name) = ‘naproxen’), returned no results. The system fed this failure back to the LLM with the prompt hints.

GraphRAG, after retry:

“The drugs that share an active ingredient with naproxen are ‘naproxen sodium’ and ‘naproxen sodium 220mg.’”

The retry switched from exact-match to CONTAINS ‘naproxen’, which caught the variants in the corpus. Without the retry loop, this question would have scored 0 for GraphRAG. Concrete evidence that the self-correction pattern earns its keep.

Example 3: where Vector RAG wins

**Question: **What is methocarbamol used for?

Vector RAG:

“Methocarbamol is indicated as an adjunct to rest, physical therapy, and other measures for the relief of discomfort associated with acute, painful musculoskeletal conditions”

GraphRAG:

“Methocarbamol is used for musculoskeletal conditions.”

Graph extraction collapsed the indication to a single canonical condition (“musculoskeletal conditions”). The richer prose version from the drug label, which is what a clinician would actually want to know & survived in the vector chunks but got lost in extraction. This is the fundamental tradeoff of structured retrieval: precision at the cost of recall, and structure at the cost of nuance.

Cost Findings

The first time I saw these numbers, I assumed I had a bug. GraphRAG is supposed to be expensive, but it makes 2–3 LLM calls per query (Cypher generation, optional retry, and answer synthesis). How can it be cheaper than Vector RAG?

The explanation is the context size. Vector RAG retrieves 5 prose chunks, each ~500 words. That’s ~2,500 tokens going into the answer-generation prompt. GraphRAG retrieves structured data in a few JSON rows, maybe 100 tokens. The answer-generation call is dramatically shorter.

Even though GraphRAG makes more LLM calls, the total tokens consumed are lower. Per query, GraphRAG is about half the cost of Vector RAG. Latency tells the opposite story. Vector RAG’s single LLM call finishes in ~1.4 seconds. GraphRAG’s serial chain (Cypher gen → run → optional retry → answer) takes ~4.6 seconds median. Hybrid pays for both pipelines plus a synthesis call, landing at ~5.6 seconds.

Based on this, if cost matters more than latency, GraphRAG is surprisingly competitive. If latency matters more than cost, Vector RAG is the only viable option for sub-2-second responses.

Where the retry loop actually fires

The 20% overall retry rate is the headline number, but it’s not evenly distributed:

Two patterns worth noting:

1. Long-tail questions are where first attempts most often miss the model relies on overly specific matching for entities it doesn’t have strong priors on. Multi-hop is where joins go wrong. The retry catches both.

2. Single-hop and aggregation have 0% retry rate. Either the first Cypher works, or the question is unanswerable (e.g., the entity isn’t in the corpus), and no amount of regeneration helps.

So, ~1 in 5 GraphRAG queries needed the self-correction pattern to succeed. That’s a real cost-benefit data point for anyone considering implementing it.

Limitations

• Single corpus, single domain: All results are on FDA drug labels. They may or may not generalize to other medical corpora (research papers, clinical notes) or to other domains entirely. The corpus also skews OTC due to openFDA’s default ordering silicea, povidone-iodine, and benzalkonium chloride are over-represented relative to what a clinician would care about.
• Small evaluation set: 30 questions isn’t a research-grade benchmark. It’s enough to see clear patterns, but individual category results are noisy at this scale (with 5 questions per category, a single judge call can swing a category result by 20 percentage points).
• LLM-as-judge bias: I used GPT-4o as the judge with a 0/1/2 rubric. LLM judges have known biases. They tend to favor longer, more structured answers, which may disproportionately advantage Hybrid. A manual spot-check confirmed reasonable agreement between judges and humans, but it isn’t a substitute for full human evaluation.
• Negation is universally weak (10–30% across systems). This is a real limitation. NOT EXISTS patterns in Cypher are syntactically tricky, and the LLM generates them inconsistently. Vector RAG can’t do negation at all. None of the three approaches handles this category well, and I don’t have a clean answer for why.
• Graph quality is capped by extraction quality: GraphRAG can only retrieve what was extracted from the labels. There is one question in the eval set — “What is benzalkonium chloride used for?” which failed for GraphRAG even after 2 retries because the extraction step didn’t capture the indication. Vector RAG retrieved it directly from the prose.
• Aggregation gold answers are partially under-specified. Several aggregation questions had gold answers like “depends on corpus,” which made LLM-as-judge scoring noisy on that category. With cleaner golds tied to actual Knowledge Graph counts, aggregation scores would likely be higher across all three systems.

Again, I’m not a clinician. This is a technical benchmark of retrieval systems, not medical advice. None of the answers should inform clinical decisions.

When to use what

Use Vector RAG when:

• Questions are mostly single-hop factual lookups.
• Your corpus is prose-heavy with low natural structure.
• Latency matters (sub-2-second responses needed).
• Cost is critical, and you can tolerate ~30% accuracy on harder questions.

Use GraphRAG when:

• You need aggregation, counting, or structured filtering.
• Multi-hop reasoning is common.
• You want explainable retrieval.
• Your data has a clear entity-relationship structure worth extracting.
• Cost matters more than latency.

Use Hybrid when:

• Accuracy is the priority, and you can afford the latency.
• Question types vary (mix of single-hop, multi-hop, aggregation, prose).
• You want a safety net when one retriever fails, the other often picks up the slack.

The honest answer to “which retrieval system should I use?” is depends on your question mix, your latency budget, and your cost budget. The point of this article is that those tradeoffs are now measurable rather than assumed. If you’re building RAG on a sufficiently structured corpus, the data here suggests running the comparison rather than defaulting to vector retrieval out of habit.

Code & Reproducibility

The full pipeline is on GitHub: data extraction, Knowledge Graph build, three RAG systems, evaluation, and analysis. Reproducing takes about 2 hours end-to-end with ~$1–2 in OpenAI usage. The README has step-by-step instructions, including the Neo4j AuraDB setup.

While AI tools assist with grammar and sentence refinement, all data, numbers, and experimental results presented here are personally verified and performed by me.