I Benchmarked 3 RAG Pipelines on 4 Datasets. GraphRAG Won — But Not How I Expected.

A live benchmark of LLM-Only, Basic RAG, and TigerGraph GraphRAG across 2,335 documents stored on TigerGraph Cloud.

Multi-hop reasoning is where RAG systems go to die.

Ask a system “Which film has the director born earlier, The Last Days of Pompeo or Border Post 58?” and suddenly flat vector search isn’t enough.

You need to traverse relationships. You need a graph.

I spent the last few weeks building a reproducible 3-pipeline benchmark to find out exactly how much a graph database backend changes the answer and where it helps most.

The Setup

Three pipelines. One fair fight.

• Pipeline 1 — LLM-Only: Pure Gemini 3.1 Flash Live with no retrieval. Just parametric memory.
• Pipeline 2 — Basic RAG: FAISS flat vector search → top-k chunks → Gemini answer.
• Pipeline 3 — GraphRAG: TigerGraph Cloud with HNSW vector search + entity graph + community retrieval → hybrid context → Gemini answer.

All three answer the same 50 questions from MultiHopRAG, each graded by GPT-4o as judge and BERTScore F1.

The data: 2,335 news articles spanning crypto fraud trials, sports, climate, politics — the full MultiHopRAG corpus — stored live on TigerGraph Cloud Enterprise 4.2.2. The graph has 3,596 document chunks with HNSW embeddings, 2,149 extracted entities (people, orgs, events), and 30 community clusters.

The Numbers

GraphRAG delivers +28 percentage points over LLM-Only and +22pp over Basic RAG. At 1.92× the cost of Basic RAG, that’s a compelling tradeoff for any application where accuracy matters.

Where the Graph Really Matters

The interesting story is in the breakdown by question type:

Temporal queries are where Basic RAG completely collapses. Questions like “Between these two articles published on different dates, which event happened first?” require the system to reason across time — something vector similarity alone can’t handle. The entity graph lets GraphRAG traverse article→entity→related-articles connections to piece together a timeline.

Why GraphRAG uses 2.3× more tokens

Basic RAG sends you the top-5 most-similar chunks. GraphRAG sends you those chunks plus entity descriptions plus community summaries. More context means more tokens (7,450 avg vs 3,219), more cost, slightly more latency — but also richer signal for the LLM to reason over.

The TigerGraph MCP Workflow

One thing that surprised me: TigerGraph ships an MCP (Model Context Protocol) server that exposes the entire graph database as Claude Code tools. I could ask Claude to create the schema, install GSQL queries, check vertex counts, and debug issues — all from the same terminal. The entity graph schema (Document→DocumentChunk→Entity→Community) was set up entirely through MCP without writing a single GSQL line by hand.

The ECC (Eventual Consistency Controller) handles the heavy lifting: it chunks documents, generates embeddings via a local MiniLM shim, calls Gemini to extract named entities, and runs Leiden clustering to form communities. All stored in TigerGraph Cloud.

The Code

Everything is open source: **github.com/UtkarshArjariya/TigerGraph-RAG**

One command boots the full stack against TG Cloud:

bash start_tgcloud.sh

The dashboard (Streamlit, port 9090) lets you run any of the 188 benchmark questions across all 3 pipelines and compare answers side-by-side.

What I’d Do Differently

1. Paid Gemini tier for entity extraction. The free tier’s 15 RPM cap means the entity graph builds in cycles overnight rather than in one shot. With billing enabled, 3,596 chunks get entities in under 5 minutes.
2. Native TigerGraph vector type. I initially used LIST<DOUBLE> for embeddings (wrong) before discovering TG 4.x has a proper VECTOR type with HNSW indexing. The add_vector_attribute MCP tool handles this cleanly.
3. Longer eval set. 50 questions gives a directional signal but ±14% confidence intervals. 200+ questions would tell a more definitive story.

Bottom Line

GraphRAG wins on accuracy, especially for the question types hardest for vector search alone. The cost is real (6.65× LLM-Only, 1.92× Basic RAG), but for any application where wrong answers have consequences, that premium buys meaningful reliability.

The graph isn’t magic — it’s structure. Structure that lets the LLM reason across relationships instead of just pattern-matching on token similarity.

All data lives on TigerGraph Cloud. Try the live demo or clone the repo