LLM Retrievers Compared: BM25, FAISS, ScaNN

Pick the right retriever for your RAG stack — lexical recall, vector speed, or both with a hybrid that actually moves the win-rate.

A practical comparison of BM25 vs FAISS vs ScaNN for LLM retrieval — when each wins, hybrid patterns, and copy-paste Python to stand them up fast.

You’ve got documents. You’ve got an LLM. And you’ve got users asking “that one line from page 43 with the weird acronym.” Retrieval is the difference between “sounds smart” and “actually helpful.” Let’s compare three workhorses — BM25, FAISS, and ScaNN — and, more importantly, how to combine them so your answers click.

Retrieval mental model (one minute)

• Lexical (BM25): matches words. Great for rare tokens, exact names, and legal citations.
• Vector (FAISS/ScaNN): matches meaning. Great for paraphrases, synonyms, and fuzzy memory.
• Hybrid: use both, then rerank. Best of both worlds for most RAG apps.

Latency budget for interactive RAG is ~100–300 ms for retrieval. Spend it wisely.

BM25: the precision scalpel

What it is: A classic ranking function from information retrieval. Scores a document by how often query terms appear, down-weighting common words and long docs.

Where it shines

• Proper nouns, IDs, codes: ERR42, “RFC 6455”, SKU strings.
• Long PDFs with headers you actually want literal matches for.
• Low-compute deployments — no GPU, tiny CPU.

Limitations

• Paraphrases slip through. “refund” ≠ “reimbursement” unless both appear.
• Stopword-heavy questions need careful tokenization.

Quick start (Python):

# pip install rank-bm25
from rank_bm25 import BM25Okapi
from nltk.tokenize import word_tokenize

docs = ["WebSocket per RFC 6455 ...", "This policy covers reimbursements ...", "ERR42 occurs when ..."]
tokenized = [word_tokenize(d.lower()) for d in docs]
bm25 = BM25Okapi(tokenized)

q = "What does RFC 6455 say about handshakes?"
scores = bm25.get_scores(word_tokenize(q.lower()))
top = sorted(zip(scores, docs), reverse=True)[:3]

FAISS: fast ANN on your laptop (and beyond)

What it is: Facebook AI Similarity Search — Approximate Nearest Neighbor indexes (flat, IVF, HNSW, PQ) with CPU/GPU options.

Where it shines

• Medium to large corpora (10k → millions) with sub-second semantic search.
• Runs anywhere: local dev, CPU VMs, or a GPU box.
• Flexible trade-offs: memory vs recall vs speed.

Limitations

• Needs embeddings that match your domain.
• Tuning index parameters is a small craft project.

Quick start (Python):

# pip install faiss-cpu sentence-transformers
import faiss, numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
vecs = model.encode(docs, normalize_embeddings=True).astype("float32")

index = faiss.IndexHNSWFlat(vecs.shape[1], 32)  # simple, strong baseline
index.hnsw.efConstruction = 200; index.hnsw.efSearch = 64
index.add(vecs)

qv = model.encode(["refund policy for travel"], normalize_embeddings=True).astype("float32")
D, I = index.search(qv, k=5)
results = [docs[i] for i in I[0]]

ScaNN: tuned for large recall at low latency

What it is: Google’s Scalable Nearest Neighbors — learned partitions + asymmetric hashing to push high recall with fewer distance computations.

Where it shines

• Millions of vectors where every millisecond matters.
• CPU-only, cloud-friendly. Often lower p95 than FAISS for the same recall on big corpora.

Limitations

• Best supported in the TensorFlow/TFDS ecosystem; extra care in pure PyTorch stacks.
• Fewer index varieties vs FAISS (but you need fewer knobs).

Quick start (Python):

# pip install scann sentence-transformers
import scann
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
emb = model.encode(docs, normalize_embeddings=True).astype(np.float32)

searcher = scann.scann_ops_pybind.builder(emb, 10, "dot_product") \
    .tree(num_leaves=200, num_leaves_to_search=80) \
    .score_ah(2, anisotropic_quantization_threshold=0.2) \
    .reorder(50).build()

qv = model.encode("travel reimbursement policy", normalize_embeddings=True).astype(np.float32)
neighbors, dists = searcher.search(qv, final_num_neighbors=5)
results = [docs[i] for i in neighbors]

Which one should you use? (field guide)

Situation Pick Rare tokens, product IDs, RFCs, error codes BM25 Small–medium corpora, general semantic queries FAISS (HNSW) Very large corpora, strict latency SLOs ScaNN Mixed queries (IDs + paraphrases) Hybrid (BM25 + Vector) You can’t tune indexes right now BM25, add vector later

Let’s be real: most production RAG benefits from hybrid. It’s the simplest way to lift answer quality without a research project.

Hybrid retrieval that actually moves the needle

Pattern: run BM25 and vector search in parallel, union results, then rerank with a cross-encoder (or your LLM).

┌─────────┐      ┌──────────┐
Q ──▶│  BM25   │──┐  ┌▶│  Vector  │
     └─────────┘  │  │ └──────────┘
                  ├──┤
                  ▼  ▼
              [Union K]
                  │
            Cross-encoder
                  │
              Top-k final

Code sketch:

# pip install sentence-transformers rank-bm25
from sentence_transformers import SentenceTransformer, CrossEncoder
from rank_bm25 import BM25Okapi
import numpy as np

# prepare
bi = SentenceTransformer("all-MiniLM-L6-v2")
cross = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

# BM25
tokens = [d.lower().split() for d in docs]
bm25 = BM25Okapi(tokens)

def retrieve(q, k=10):
    # lexical
    bm_scores = bm25.get_scores(q.lower().split())
    top_bm = np.argsort(bm_scores)[::-1][:k]

    # vector (faiss or scann; here brute-force for brevity)
    emb = bi.encode(docs, normalize_embeddings=True)
    qv  = bi.encode([q], normalize_embeddings=True)
    sims = (emb @ qv.T).ravel()
    top_vec = np.argsort(sims)[::-1][:k]

    # union
    cand_idx = list(dict.fromkeys(list(top_bm) + list(top_vec)))  # dedupe preserve order
    pairs = [[q, docs[i]] for i in cand_idx]
    rerank = cross.predict(pairs)
    top = [docs[i] for i in np.argsort(rerank)[::-1][:5]]
    return top

Why it works: BM25 rescues acronyms, IDs, and exact phrases; vectors catch paraphrases; reranking chooses the actually relevant chunk.

Metrics that matter (and how to improve them)

• Recall@k: chance the gold doc is in your top-k before reranking. Boost by increasing k or improving the index (more leaves / efSearch).
• MRR / nDCG: ranking quality after reranking. Boost by training the cross-encoder on your domain or using the LLM as a judge with strict prompts.
• Latency p95: user experience. Cap k and do early fusion (smaller unions) to keep p95 under budget.

A simple rule: start with k=8 BM25 + k=8 vector, rerank to 5. If latency is high, drop to 6+6. If recall is low, raise vector k first.

Practical edges and gotchas

• Chunking: too big → low resolution; too small → context lost. Start at 300–500 tokens with overlap 50–100.
• Embeddings: pick one with strong vocabulary for your domain; normalize vectors.
• Multilingual: BM25 language matters; consider language-aware analyzers or multilingual embeddings.
• Freshness: re-embed on updates; FAISS/ScaNN support incremental adds (periodic rebuilds still healthy).
• Caching: memoize query → results for hot paths; it’s free latency.

Tiny deployment map (keep it boring)

[Ingest] -> [Chunk+Embed] -> [Index: FAISS/ScaNN]
                      └-> [Lexical: BM25/Elastic]
[Query]  -> Parallel {BM25, Vector} -> [Rerank] -> [Prompt LLM]

Store provenance (doc_id, page, header) with each chunk. Your LLM can cite correctly, and your debugger will thank you.

Wrap-up

BM25, FAISS, and ScaNN aren’t rivals; they’re tools. BM25 gives you crisp lexical recall. FAISS gives you flexible, fast semantic search. ScaNN pushes large-scale latency down without breaking recall. Most teams win by hybriding and reranking. Start simple: BM25 + FAISS, k=8 each, rerank to 5. Measure, then tune.