Vectorless RAG Won’t Save Your Agent. But It Reveals What’s Actually Broken.

A new retrieval approach claims 98.7% accuracy by throwing out vector databases entirely. The number is real, the idea is genuinely good — and both are being sold to you with an asterisk nobody’s reading aloud.

A headline has been ricocheting around AI circles for months: a retrieval method that scores 98.7% accuracy on hard financial documents — where the traditional approach scores around 50% — by deleting the vector database entirely. No embeddings, no chunking, no similarity search. The thing every RAG tutorial for three years told you was mandatory, thrown in the bin, and accuracy nearly doubled.

If you build agents, that’s a spicy claim, because retrieval is often where your agent quietly goes wrong. So I spent time with the primary sources instead of the hype. My conclusion is split in two, and both halves matter: the core idea is genuinely good and worth understanding — and the number selling it comes with an asterisk that changes what you should do about it. Let’s take both seriously.

First, the plain-language version of both RAGs

Quick grounding, no jargon assumed. RAG — retrieval-augmented generation — just means: before an AI answers a question, it first looks something up and reads the relevant text. It’s how an agent answers questions about your specific documents instead of making things up from memory. The “retrieval” step is the looking-up part, and it’s where quality is won or lost.

For years, retrieval has meant vector search. You chop documents into chunks, convert each chunk into a list of numbers (an embedding) that captures its meaning, and at question time you find the chunks whose numbers are mathematically closest to the question. Closest-in-meaning, roughly.

Vectorless RAG throws that out. Instead of measuring similarity, it hands an AI the document’s structure — its table of contents, its section tree — and lets the model reason its way to the right section, the way you’d flip to Chapter 4 because you know that’s where the answer lives. The best-known version, an open-source framework called PageIndex, borrows the idea from AlphaGo: don’t search everything exhaustively, navigate intelligently.

Here’s that navigation in miniature — a toy version that reasons down a document tree to the right page, with no vectors anywhere:

def navigate(question, node, path=None):
    path = path or []
    kids = node.get("sections", [])
    if not kids:                       # leaf: answer location reached
        return path + [node["title"]], node.get("page")
    choice = llm_pick(question, kids)  # LLM reasons: which branch holds the answer?
    return navigate(question, kids[choice], path + [node["title"]])

Ask it “what was the change in net revenue from Q2 to Q3?” and it walks Annual Report > Financial Statements > Income Statement and lands on the right page — by reasoning about structure, not by measuring numerical closeness. That's the whole philosophy in six lines.

Why the idea is actually good (and why it’s an agent story)

Strip away the hype and vectorless RAG is pointing at a real, deep flaw in vector search — one that regular readers will recognize instantly.

Similarity is not relevance. Ask “what was the change in net revenue from Q2 to Q3?” and the chunks most similar to that sentence are other sentences that talk about net revenue and quarters — not necessarily the two specific numbers you need to subtract. Vector search retrieves things that sound like the question. What you actually needed was the thing that answers it. Those come apart exactly when the question is hard.

Vector search finds text that sounds like your question. You needed the text that answers it. On hard questions, those are different paragraphs.

Similarity vs. relevance, side by side. Asked for a Q2→Q3 revenue change, vector search (left) returns the chunks that sound most like the question — and misses the two numbers you needed. Structural reasoning (right) walks the document tree to the income statement where the numbers actually live. On hard, structured documents, “closest in meaning” and “contains the answer” are different places.

And here’s why this is really an agent article in disguise. If you’ve read my piece on context engineering, you know an agent is only as good as what lands in its context window — feed it the wrong three chunks and even a brilliant model produces a confident, wrong answer.

Retrieval quality is context quality. A bad retriever is just a machine for poisoning your agent’s context automatically, at scale. Vectorless RAG is one attempt to poison it less — which is why it’s worth your attention even if you never touch a vector database again.

Now read the asterisk

Here’s where the production-skeptic hat goes on, because the 98.7% is doing a lot of unspoken work.

It’s a vendor’s own number. The 98.7% comes from VectifyAI, the company behind PageIndex, evaluating its own commercial system on its own published benchmark run. That doesn’t make it false — but a vendor reporting a near-perfect score for its own product is a claim to verify, not a fact to build on. Notably, VectifyAI’s own benchmark repository even cautions that the test set’s “correct” answers may themselves contain errors and ambiguities — a striking admission that the ground truth isn’t gospel.

The number hides the costs that decide production. One sharp critique noted that behind the accuracy figure, there is essentially zero published data on latency (how long a query takes), throughput, or cost per query. And those costs aren’t incidental here — they’re structural.

Remember how it works: it doesn’t do one fast math lookup, it makes repeated calls to the language model to reason down the tree. As one commenter put it, it doesn’t eliminate dependencies so much as swap vector approximation for LLM-reasoning approximation — and model calls are slower and pricier than a similarity lookup by a wide margin. You may well be trading dollars and seconds for those accuracy points.

A benchmark that reports accuracy but hides latency and cost isn’t a result. It’s an advertisement with a number on it.

The 50% baseline is convenient. “Vector RAG only gets ~50%” refers to hard, structured financial documents — the exact terrain vectorless is built to win. It is not a claim that vector search is 50% at everything. On large-scale, multi-document search where speed matters, vectors still win comfortably. The eye-popping gap is real and cherry-picked to the home field.

Make the trade-off concrete. A vector lookup is essentially one fast math operation over pre-computed numbers — milliseconds, fractions of a cent. Reasoning down a document tree can mean five, ten, or more sequential model calls, each adding latency and cost, before the agent even starts composing its answer.

For a one-off analysis of a critical contract, that’s a bargain. For an agent fielding thousands of user questions an hour, it can be the difference between a viable product and a runaway bill. The right answer depends entirely on your volume and your tolerance for wrong answers — which is exactly why the vendor’s single number can’t decide it for you.

The pattern here is bigger than RAG

Step back, because this is the part that outlives PageIndex specifically.

A near-perfect benchmark score, published by the vendor, on the vendor’s chosen benchmark, with the operational costs conveniently absent — that’s a shape you will see again and again in agent tooling. It’s the same trap I wrote about in my verifiability piece: a single impressive number tells you almost nothing about whether a system works in your conditions, and the more dazzling the number, the more skeptically you should read the setup that produced it.

This doesn’t make PageIndex bad — by several independent accounts it genuinely shines on long, structured, high-stakes documents. It makes the marketing incomplete. And knowing the difference between a good tool and a good advertisement for that tool is most of what separates production engineers from demo-watchers.

What to actually do with this

So should you rip out your vector database this weekend? Almost certainly not. Here’s the honest decision guide:

• Match the tool to the document, not the hype. Long, structured, high-stakes docs (financial filings, contracts, medical records) where exact answers matter and structure is rich → vectorless reasoning is worth a genuine trial. Large-scale search across millions of short, messy documents where speed and cost dominate → vectors still win.
• Assume hybrid is the endgame. Even the write-ups praising vectorless expect production systems to combine both — vectors for broad recall, structural reasoning for precision. “Vectorless” is a useful retrieval option to add, not a religion to convert to.
• Benchmark on your documents, with cost and latency measured. The 98.7% was earned on someone else’s data with the price tag hidden. Your only trustworthy number is the one you produce on your own documents, timing and costing every query.
• Fix retrieval as a context problem. Whatever method you pick, the goal is the same: get the right text into your agent’s window. That’s the metric. The retrieval technique is just means to it.

The takeaway that’s actually worth keeping

Vectorless RAG is a genuinely clever idea wrapped in a genuinely oversold number, and both facts are true at once. The idea — that relevance beats similarity, that reasoning over structure can beat measuring closeness — is a real contribution, and it sharpens how you should think about feeding any agent. The number — 98.7%, no context, no cost — is a lesson in reading benchmarks like an adult.

If this piece leaves you with one durable habit, let it be this: when a tool promises to fix your agent with a single spectacular statistic, the most valuable thing you can do is find the asterisk before you find the install command.

The best retrieval upgrade isn’t a new database. It’s the discipline to ask what every impressive number left out.

Follow Think in AI Agents to catch it. And tell me in the comments: what’s the most oversold benchmark number you’ve been pitched this year? Best example gets dissected in a future piece.