Stop Using Embeddings for Everything: Building Faster and More Explainable RAG with PageIndex and BM25

How BM25, Lucene, PageIndex, and structured retrieval can outperform expensive embedding pipelines for most enterprise document search problems — and when they absolutely should.

Introduction

Every AI team building a document Q&A system reaches the same junction. Someone opens a laptop, shares a slide about RAG architecture, and before long the whiteboard is covered in boxes: chunking → embedding model → vector database → similarity search → LLM. It feels complete. It feels modern. And for many teams, it becomes a $40,000-per-year infrastructure decision made in a 45-minute architecture meeting.

But here’s the uncomfortable truth that most AI practitioners only discover after six months in production: for a significant class of enterprise document problems, you never needed a vector database at all.

This is not a contrarian take for its own sake. It is a hard-won lesson from real deployments: HR policy assistants returning wrong policy versions because embeddings couldn’t distinguish document dates, insurance claim systems hallucinating policy numbers because semantic similarity matched the wrong coverage clause, legal document search tools that couldn’t find an exact clause because the embedding model paraphrased it into oblivion.

This article is a complete technical guide to Vector-less RAG — what it is, how it works internally, when it outperforms embedding-based approaches, and how to build it in production using .NET 10, Lucene.NET, and Gemini.

The Real-World Problem Statement

Imagine you are a Tech Lead at a mid-size insurance company. Your team has been tasked with building an internal AI assistant that answers employee questions about:

• HR policies (leave, reimbursement, appraisal)
• Insurance claim documents
• Compliance SOPs
• Internal knowledge base articles

The initial vector RAG prototype works reasonably well during demos. But then you hit production:

Problem 1 — Stale embeddings: When the HR team updates a leave policy, the old chunks remain indexed in the vector database. You now have ghost embeddings — semantically similar but factually wrong.

Problem 2 — Exact keyword failures: An employee asks, “What is the reimbursement limit for Business Class flights for VP-level employees?” The embedding model, trying to be semantically smart, retrieves a paragraph about economy class reimbursement that scores highly because it also talks about flights and reimbursement limits.

Problem 3 — Escalating costs: Your embedding API bill is growing proportionally with every new document upload. You have 50,000 policy documents and growing.

Problem 4 — Chunking nightmares: A complex insurance policy PDF is split mid-sentence across chunks. The retrieved chunk references “the above table,” but the table is in the previous chunk. The LLM hallucinates the table’s contents.

This is not a hypothetical scenario. These are the exact failure patterns that push mature engineering teams away from pure vector RAG toward structured, keyword-aware retrieval systems.

Why Traditional RAG Became Popular

Retrieval Augmented Generation arrived at the perfect moment. Large language models had become extraordinarily capable at synthesis and reasoning, but they had a critical flaw: their knowledge was frozen at training time. RAG solved this elegantly — instead of fine-tuning a model on your proprietary data (expensive, risky, slow), you retrieve the relevant context at query time and inject it into the prompt.

The early academic papers demonstrated impressive results on open-domain QA benchmarks. The technique required:

• A way to store and search documents
• A way to retrieve the most relevant fragments
• A way to pass those fragments to an LLM

Vector embeddings became the default storage and retrieval mechanism because they could capture semantic similarity in ways that keyword search couldn’t. “What are the working hours?” and “What time does the office open?” mean the same thing, and embeddings could connect them. This was genuinely magical and genuinely useful.

But what works on a research benchmark isn’t always what works in an enterprise HR portal.

Traditional RAG Architecture

┌─────────────────────────────────────────────────────────┐
│                  TRADITIONAL VECTOR RAG                 │
│                                                         │
│  INDEXING PIPELINE                                      │
│  ─────────────────                                      │
│  PDF / DOCX / HTML                                      │
│       │                                                 │
│       ▼                                                 │
│  [ Text Extraction ]  ← (PDF parser, OCR if needed)     │
│       │                                                 │
│       ▼                                                 │
│  [ Chunking ]  ← Split into 512-token windows           │
│       │                                                 │
│       ▼                                                 │
│  [ Embedding Model ]  ← OpenAI ada-002 / BERT           │
│       │                                                 │
│       ▼                                                 │
│  [ Vector Database ]  ← Pinecone / Qdrant / Weaviate    │
│       │                                                 │
│       ▼                                                 │
│  [ Dense Index ]  ← 1536-dim float vectors              │
│                                                         │
│  QUERY PIPELINE                                         │
│  ──────────────                                         │
│  User Question                                          │
│       │                                                 │
│       ▼                                                 │
│  [ Embed Question ]  ← Same embedding model             │
│       │                                                 │
│       ▼                                                 │
│  [ Similarity Search ]  ← Cosine / dot-product          │
│       │                                                 │
│       ▼                                                 │
│  [ Top-K Chunks ]  ← Retrieved fragments                │
│       │                                                 │
│       ▼                                                 │
│  [ Prompt Builder ]  ← Inject chunks into system prompt │
│       │                                                 │
│       ▼                                                 │
│  [ LLM ]  ← Gemini / GPT-4 / Claude                     │
│       │                                                 │
│       ▼                                                 │
│  [ Answer + Citation ]                                  │
└─────────────────────────────────────────────────────────┘

What Happens Internally in Embedding-Based RAG

Understanding the internals exposes both the power and the fragility of traditional RAG.

Chunking Process

Chunking is the process of splitting a large document into smaller, indexable fragments. A typical enterprise document — a 120-page HR policy manual — cannot be fed entirely into an LLM context window, and even if it could, the LLM would struggle to focus on the relevant section. So the document is split.

Common strategies:

• Fixed-size chunking: Split every N tokens with M tokens of overlap. Simple. Brutal. Loses structure entirely.
• Sentence-based chunking: Split at sentence boundaries. Better, but a single sentence often lacks context.
• Paragraph-based chunking: More semantically coherent but inconsistent in length.
• Recursive character splitting: LangChain’s default — tries multiple separators in order.

The fundamental problem with all chunking strategies is that documents have structure that chunking destroys. A table in a PDF is split across chunks. A numbered list is fragmented. The context of “Section 3.4, subsection 2” is lost entirely. The chunk has no idea where it came from in the document’s hierarchy.

Embedding Generation

Once chunks exist, each one is passed through an embedding model. OpenAI’s text-embedding-ada-002 outputs a 1,536-dimensional floating-point vector. text-embedding-3-large outputs 3,072 dimensions. These vectors capture the semantic "meaning" of the chunk in a high-dimensional geometric space.

The cost is real: at $0.0001 per 1,000 tokens for ada-002, a corpus of 10 million tokens (roughly 1,000 average enterprise documents) costs $1 per full index rebuild. But at enterprise scale — 500,000 documents, frequent updates — this becomes significant, particularly when combined with re-indexing costs.

Vector Database and Similarity Search

The generated vectors are stored in a vector database (Pinecone, Qdrant, Weaviate, pgvector, Azure AI Search with vector fields). At query time, the user’s question is embedded using the same model, and an Approximate Nearest Neighbor (ANN) search finds the top-K chunks whose vectors are closest to the question vector in cosine or dot-product space.

This is genuinely powerful for semantic matching. It is genuinely poor for:

• Exact keyword retrieval
• Boolean logic (“find documents where department = ‘HR’ AND effective_date > ‘2024–01–01’”)
• Document-structure-aware retrieval
• Queries involving specific names, numbers, codes, or IDs

Vector RAG Limitations in Enterprise Context

Embedding Cost at Scale

Consider a large bank with 2 million internal policy, procedure, and compliance documents, averaging 5,000 tokens each. Full corpus: 10 billion tokens. At OpenAI pricing, initial indexing alone costs ~$1,000. But in enterprise environments, documents change continuously. Each change triggers re-embedding of affected chunks. At 10% document churn per month, you are spending $100/month just on re-embedding — plus vector database hosting fees on top.

For organizations using Azure OpenAI at reserved capacity, this is manageable. For smaller enterprises or teams with tight budgets, this is a blocker.

Re-indexing Challenges

When a policy document is updated, you cannot simply “update” its embedding. The document must be re-chunked, re-embedded chunk by chunk, and the old vectors must be deleted and replaced. In a naive implementation, you end up with ghost vectors — outdated chunks that still score high for certain queries because their semantic content is similar to the new version, but their facts are wrong.

Enterprise example: An insurance company updated its motor claim settlement policy. The new policy raised the no-claim discount from 15% to 20%. The old chunks — with “15%” — were not fully purged during the hasty re-indexing. For three weeks, the AI assistant cited the wrong discount percentage to customers. The bug was invisible at the vector level because both chunks were semantically identical.

Chunking Problems and Lost Context

The most insidious failure mode in production RAG. A 200-page insurance policy is chunked into 800 fragments. Chunk #342 reads: “The deductible applies as specified in Table 7 above.” But Table 7 is in Chunk #289. When Chunk #342 is retrieved without Chunk #289, the LLM cannot see Table 7. It either hedges, hallucinates the table’s values, or ignores the reference entirely.

This is not a chunking parameter tuning problem. It is a fundamental structural problem with treating documents as flat text streams.

Exact Keyword Failures

Semantic search excels at conceptual matching but fails at exact matching. If an employee asks “What is the reimbursement limit under policy code HR-EXP-2024–07B?”, the embedding model has no way to preferentially match the exact policy code. It will retrieve semantically similar expense-related chunks that may or may not include that specific policy.

SQL databases solve this trivially. Elasticsearch and Lucene solve this trivially. Vector databases do not.

Hallucination Risks from Poor Retrieval

Garbage in, garbage out. If the retrieved chunks are wrong — outdated, incomplete, or semantically adjacent but factually different — the LLM will confidently synthesize an answer from bad context. The LLM cannot know that the retrieved chunk is wrong. It trusts what it’s given. This is the single largest source of hallucination in production RAG systems.

Vector-less RAG: Definition and Core Concepts

Vector-less RAG (also called BM25 RAG, Keyword RAG, or Structured RAG) is a Retrieval Augmented Generation architecture that replaces dense vector embeddings with classical information retrieval techniques — primarily full-text indexing, BM25 scoring, inverted indexes, and structured metadata search — to identify and retrieve relevant document passages at query time.

The key philosophical difference: instead of asking “what documents are semantically similar to this question?”, vector-less RAG asks “what documents contain the specific terms, patterns, metadata, and structural context that match this query?”

For a large class of enterprise problems, this question is more precise, more auditable, and more correct.

Core Concepts

Full-Text Indexing is the process of parsing every word in every document and building an inverted index — a lookup table that maps every term to the list of documents (and positions within those documents) that contain it. This is the foundation of search engines from Apache Lucene to Elasticsearch to SQL Server Full-Text Search.

Inverted Index is the data structure at the heart of full-text search:

┌────────────────────────────────────────────────────┐
│              INVERTED INDEX STRUCTURE              │
│                                                    │
│  Term          → Document List (with positions)    │
│  ──────────────────────────────────────────────    │
│  "reimbursement" → [Doc3:p2, Doc7:p5, Doc12:p1]    │
│  "business class" → [Doc3:p2, Doc19:p4]            │
│  "VP-level"      → [Doc3:p2, Doc7:p8]              │
│  "2024"          → [Doc1,Doc3,Doc7,Doc11...]       │
│                                                    │
│  Query: "business class reimbursement VP-level"    │
│  → Intersect document lists                        │
│  → Rank by BM25                                    │
│  → Return Doc3, page 2                             │
└────────────────────────────────────────────────────┘

BM25 (Best Match 25) is a probabilistic ranking function that scores documents by term frequency, inverse document frequency, and document length normalization. It is the industry standard for keyword relevance ranking and is used internally by Elasticsearch, Lucene, and most enterprise search platforms.

The BM25 score for a document D given query Q is:

Score(D, Q) = Σ IDF(qi) × [ tf(qi, D) × (k1 + 1) ]
                              [ tf(qi, D) + k1 × (1 - b + b × |D|/avgdl) ]
Where:
  IDF(qi) = log((N - n(qi) + 0.5) / (n(qi) + 0.5) + 1)
  tf      = term frequency in document
  k1      = term frequency saturation (typically 1.2–2.0)
  b       = length normalization (typically 0.75)
  |D|     = document length
  avgdl   = average document length in corpus

BM25 naturally handles exact terms, penalizes term stuffing via saturation, and rewards term rarity via IDF. It is extremely fast (millisecond-range even on millions of documents) and has zero per-query infrastructure cost beyond the search index itself.

The PageIndex Concept

PageIndex is the structural innovation that makes vector-less RAG practical for enterprise document retrieval. Instead of chunking documents into arbitrary token windows, PageIndex treats each page of a document as a first-class, independently retrievable unit, preserving its original structure, layout, and positional context.

Why PageIndex Was Introduced

Chunking was invented as a compromise: LLMs have limited context windows, so you can’t pass in a whole document, so you split it. But chunking destroys the natural unit of human documents — the page. Business documents are written, formatted, reviewed, and cited by page number. “See page 12 of the reimbursement policy” is meaningful. “See chunk 342 of the reimbursement policy” is not.

PageIndex restores pages as the retrieval unit, making citations auditable, context complete (within a page), and structure-preserving.

PageIndex vs Chunking

CHUNKING APPROACH                   PAGEINDEX APPROACH
──────────────────                  ──────────────────
Text extracted as flat stream       Text extracted per page
Split at token boundary             Page boundary = natural split
No structural metadata              Page number, section, heading stored
Table split across chunks           Table stays within page record
Citation: impossible to generate    Citation: "Policy Manual, Page 7"
Re-index: all chunks invalidated    Re-index: only changed pages
Context: fragment of document       Context: complete page unit

PageIndex Internal Architecture

┌──────────────────────────────────────────────────────────────┐
│                    PAGEINDEX PIPELINE                        │
│                                                              │
│  PDF Document                                                │
│       │                                                      │
│       ▼                                                      │
│  [ PDF Parser ]  ← PdfPig / iText / Tesseract (OCR)          │
│       │                                                      │
│       ▼                                                      │
│  ┌────────────────────────────────────┐                      │
│  │  Per-Page Extraction               │                      │
│  │  ─────────────────                 │                      │
│  │  PageNumber: 7                     │                      │
│  │  DocumentId: "HR-POLICY-2024"      │                      │
│  │  Title: "Business Travel Policy"   │                      │
│  │  Section: "3.4 Air Travel"         │                      │
│  │  Text: "VP-level employees are..." │                      │
│  │  HasTable: true                    │                      │
│  │  HasImage: false                   │                      │
│  └────────────────────────────────────┘                      │
│       │                                                      │
│       ▼                                                      │
│  [ Metadata Enrichment ]  ← Department, date, version        │
│       │                                                      │
│       ▼                                                      │
│  [ Lucene Index ]  ← Full-text + structured fields           │
│       │                                                      │
│       ▼                                                      │
│  [ PageRecord in SQL ]  ← Persistent store with all metadata │
│                                                              │
│  QUERY FLOW                                                  │
│  ──────────                                                  │
│  User: "Business class reimbursement for VP employees?"      │
│       │                                                      │
│       ▼                                                      │
│  [ Query Parser ]  ← Tokenize, expand, filter                │
│       │                                                      │
│       ▼                                                      │
│  [ BM25 Search on Lucene ]  ← Score all matching pages       │
│       │                                                      │
│       ▼                                                      │
│  [ Top-K Pages ]  ← e.g. HR-POLICY-2024, Pages 7, 8, 12      │
│       │                                                      │
│       ▼                                                      │
│  [ Context Builder ]  ← Assemble page texts into prompt      │
│       │                                                      │
│       ▼                                                      │
│  [ LLM Answer ]  ← Answer with citations: "Page 7 states..." │
└──────────────────────────────────────────────────────────────┘

Detailed Comparison Tables

1. Traditional RAG vs Vector-less RAG

2. Chunking vs PageIndex

3. BM25 vs Embeddings

Real-World Use Cases

HR Policy Assistant

An HR policy assistant built on vector RAG repeatedly fails when employees ask about specific policy codes, effective dates, or exact allowance amounts. The semantic model retrieves the “closest” policy but not the correct one.

Vector-less RAG wins here because:

• Policies are structured documents with well-defined sections
• Employees ask exact questions (“What is the leave encashment limit for CL?”)
• Dates and version numbers matter — the latest policy must be retrieved, not the semantically closest
• Citation of specific policy page builds employee trust

PageIndex approach: Each policy PDF is indexed per page. BM25 retrieves the specific page containing “casual leave encashment” with the correct effective date. The LLM synthesizes a precise answer with the citation “HR Leave Policy 2024, Page 4.”

Insurance Documents

Insurance policies contain specific tables of coverage limits, exclusion clauses, and claim procedures. A user asking “Is damage due to flooding covered under my motor policy?” requires exact clause retrieval, not semantic approximation.

Vector-less RAG with BM25 naturally surfaces clauses containing “flood,” “motor,” and “coverage” — and their exact exclusion conditions. A vector model might retrieve a semantically similar clause from a home insurance policy.

Legal Contracts

Legal document search has zero tolerance for semantic approximation. “The indemnification clause” means a specific clause, not something semantically similar to indemnification. Attorneys need exact text, with page and section citation, for court submissions.

PageIndex + BM25 provides exactly this: page-precise retrieval with section-level metadata and verbatim context for LLM synthesis.

Compliance and SOP Search

Compliance teams querying SOPs often use regulatory codes, procedure IDs, and standard names that must match exactly. “Show me the procedure for ISO-27001 access control review” requires BM25’s exact term matching, not semantic proximity.

Hybrid RAG: The Enterprise Standard

The most sophisticated production RAG systems don’t choose between keywords and semantics — they use both, plus reranking, creating a pipeline that gets the best of all retrieval strategies.

┌─────────────────────────────────────────────────────────────┐
│                      HYBRID RAG PIPELINE                    │
│                                                             │
│  User Query                                                 │
│       │                                                     │
│       ├─────────────────────────────┐                       │
│       │                             │                       │
│       ▼                             ▼                       │
│  [ BM25 / Lucene ]          [ Embedding Model ]             │
│  Keyword matches            Semantic matches                │
│       │                             │                       │
│       ▼                             ▼                       │
│  [ PageIndex Results ]      [ Vector DB Results ]           │
│       │                             │                       │
│       └──────────┬──────────────────┘                       │
│                  ▼                                          │
│           [ Fusion / RRF ]  ← Reciprocal Rank Fusion        │
│                  │                                          │
│                  ▼                                          │
│           [ Reranker ]  ← Cross-encoder (Cohere / local)    │
│                  │                                          │
│                  ▼                                          │
│           [ Top-K Pages ]  ← High-confidence, ranked        │
│                  │                                          │
│                  ▼                                          │
│           [ Metadata Filter ]  ← Department, date, version  │
│                  │                                          │
│                  ▼                                          │
│           [ Prompt Builder ]                                │
│                  │                                          │
│                  ▼                                          │
│           [ LLM ] → Answer + Citations + Confidence Score   │
└─────────────────────────────────────────────────────────────┘

Reciprocal Rank Fusion (RRF) combines result lists from multiple retrieval systems without requiring score normalization: RRF(d) = Σ 1/(k + rank_i(d)) where k is typically 60. This robustly merges BM25 and vector rankings without needing calibrated scores from each system.

Hybrid RAG is the enterprise standard because:

• BM25 handles exact terms, IDs, and structured queries perfectly
• Embeddings handle paraphrasing, synonyms, and conceptual queries
• Reranking eliminates false positives from both systems
• The result set is both precise and semantically aware

Production Best Practices

Document Preprocessing: Normalize whitespace, remove headers/footers that repeat on every page, detect and extract tables separately from body text, handle multi-column layouts before indexing.

OCR Handling: Use Tesseract with language-specific models. Store confidence scores per page — pages below 70% OCR confidence should be flagged for manual review before being served to users.

Search Optimization: Use field-boosting in Lucene — boost matches in document titles and section headings vs body text. Add query expansion for common abbreviations (EL → Earned Leave, CL → Casual Leave).

Security: Implement document-level access control in SQL. Filter search results against the user’s permission set before passing to LLM. Never pass unauthorized documents into the context window.

Monitoring: Track retrieval precision (did the right page come back?) separately from answer quality. Log every query, retrieved page set, and LLM response for audit purposes.

Caching: Cache BM25 results for common queries. Cache LLM responses for identical (query + context) pairs. Use Redis with TTL aligned to document update frequency.

Key Insights

Common Mistakes:

1. Defaulting to vector RAG without evaluating whether your query patterns actually need semantic search. Run a sample of 50 real user queries. How many require semantic understanding vs exact term matching? The answer often surprises teams.
2. Ignoring document structure. Treating a PDF as a flat text stream is the root cause of 60% of enterprise RAG failures.
3. Not building citation infrastructure from day one. Users need to verify answers. “The AI said so” is not acceptable in legal, compliance, or HR contexts.
4. Using chunking without overlap testing. Overlap adds retrieval redundancy but inflates index size and can retrieve duplicate context.

Architecture Decisions:

• Start with BM25 + PageIndex. Add embeddings only when you identify queries that genuinely fail without semantic understanding.
• Keep the search layer and the LLM layer cleanly separated. Your search strategy will evolve; your LLM will change. Neither should know about the other.
• Design for auditability from day one. Every answer must trace back to a specific page in a specific document version.

Conclusion

The choice between vector RAG, vector-less RAG, and hybrid RAG is not a technology preference — it is an architectural decision that must be driven by your query patterns, your document corpus characteristics, and your production constraints.

Use vector-less RAG (BM25 + PageIndex) when:

• Your queries use exact terms, IDs, policy codes, or specific names
• Your documents are structured (policies, contracts, SOPs, manuals)
• You need auditable, page-precise citations
• Cost and infrastructure simplicity are priorities
• You are operating in regulated industries requiring explainable retrieval

Use embedding-based RAG when:

• Your queries are conversational and paraphrase-heavy
• Your documents are unstructured narratives (news, essays, transcripts)
• Semantic proximity matters more than exact term matching
• You have the infrastructure budget and the corpus is relatively stable

Use hybrid RAG when:

• You have a mix of query types (exact + semantic)
• You need the highest possible retrieval precision for enterprise deployment
• You can afford the additional infrastructure complexity
• You are building a production system that must serve diverse user populations

For most enterprise document search problems — HR policies, insurance documents, legal contracts, compliance SOPs — vector-less RAG is not a compromise. It is the right tool for the job. Start there. Add semantics when you have evidence that users need it.