Understanding Hybrid Search: Combining BM25 and Vector Search for Better Retrieval

Understanding Hybrid Search: Combining BM25 and Vector Search for Better Retrieval

Introduction

In today’s world of massive data and complex queries, traditional search methods fall short. Users don’t always know the exact keywords to search for, and systems struggle to understand intent. Hybrid search solves this problem by combining two powerful approaches: keyword search (for precision) and vector search (for understanding).

Let me walk you through how this works, with a real-world insurance company example that you can apply to any industry.

The Problem with Single Search Approaches

Imagine you work at an insurance company’s customer support desk:

• Scenario 1: A customer calls: “My policy number is POL123456.”

• Traditional keyword search works perfectly here. It finds the exact policy instantly.

-Scenario 2: Another customer calls: “I paid my instalments last week, but it’s not showing in the system.”

• Keyword search struggles. The exact terms “payment not updated” might not exist in your documents.

• You need semantic understanding to recognize this is a payment delay issue.

The solution? Don’t choose between keyword and vector search. Use both simultaneously. That’s hybrid search.

Part 1: Understanding Keyword Search with BM25

What is BM25?

BM25 (Best Matching 25) is an algorithm that ranks documents based on how well they match search terms. It’s widely used in production systems like Elasticsearch because it’s fast, reliable, and effective.

How BM25 Works

1. Tokenization: The document “My policy number is POL123456” gets split into words: [“My”, “policy”, “number”, “is”, “POL123456”]

2. Inverted Index: The system creates a map:

“policy” → [doc1, doc5, doc12]

“number” → [doc1, doc3, doc8]

“POL123456” → [doc1]

3. Scoring: For a query “policy number POL123456”:

• Documents containing all three terms rank highest

• Documents containing rare terms (like “POL123456”) get higher scores

• Term frequency matters — documents mentioning “policy” 5 times score higher than mentioning it once

4. Final Ranking: Top matching documents are returned, sorted by relevance score

BM25 Advantages & Limitations

✅ Advantages:

• Extremely fast for exact phrase matching

• Perfect for structured data (IDs, names, numbers)

• Works without training or expensive computations

• Handles typos and spelling variations well

• Efficient with large datasets

❌ Limitations:

• Doesn’t understand synonyms (searching “payment” won’t find “transaction”)

• Can’t grasp context or intent

• Struggles with semantically similar but different wording

• Returns irrelevant results if exact terms are missing

Data Storage for Keyword Search

Data is stored in an inverted index format:

• Documents are broken into words

• Words are mapped to the documents containing them

• Original text remains as strings for display

• Updated in real-time as new data arrives

Example Storage:

Index: {

“policy”: {doc1, doc5, doc12},

“number”: {doc1, doc3, doc8},

“POL123456”: {doc1},

“instalments”: {doc2, doc5, doc9},

“paid”: {doc2, doc7, doc11}

}

Part 2: Understanding Vector Search with Embeddings

What is Vector Search?

Vector search converts text into dense numerical vectors (embeddings) and finds documents by calculating similarity between vectors. It understands meaning, context, and intent.

How Vector Search Works

1. Embedding Model: Uses pre-trained models (OpenAI, Hugging Face, or custom ones) to convert text into vectors

Text: “I paid my instalments”

↓ (via embedding model)

Vector: [0.12, -0.45, 0.89, …, 0.34] (dimensions: 1536 or more)

2. Vector Storage: These vectors are stored in specialized vector databases:

• Pinecone

• Weaviate

• ChromaDB

• Milvus

3. Similarity Calculation: When a query comes in, it’s converted to a vector too:

Query: “Why isn’t my payment showing?”

↓ (via same embedding model)

Query Vector: [0.11, -0.43, 0.91, …, 0.36]

4. Cosine Similarity: The system calculates how similar the query vector is to document vectors:

Similarity =

(Query Vector · Documet Vector) / (||Query Vector|| × ||Document Vector||)

Range: 0 to 1 (1 = identical, 0 = completely different)

5. Ranking: Documents with highest similarity scores are returned

What Makes Vector Search Powerful

Semantic Understanding:

• Query: “payment didn’t go through”

• Finds: Documents about payment failures, transaction issues, processing delays

• Even if exact words don’t match, meaning is captured

Context Awareness:

• “I need help with my insurance” understands it’s a support request

• “What’s the premium rate?” knows it’s asking about costs

• Different intents, recognized correctly

Synonym Handling:

• “transaction” ≈ “payment” (similar vectors)

• “premium” ≈ “cost” (similar vectors)

• Variations are automatically understood

Vector Search Advantages & Limitations

✅ Advantages:

• Understands meaning and context

• Finds semantically similar documents

• Handles synonyms and variations naturally

• Works with fuzzy or incomplete queries

• Excellent for complex, natural language questions

❌ Limitations:

• Slower than keyword search (computing similarity takes time)

• Requires embedding models (computational cost)

• Can be less precise for exact matches

• May return false positives (semantically similar but irrelevant)

• Requires quality training data for good embeddings

Part 3: The Complete Hybrid Search Architecture

Now, let’s combine both approaches into a powerful hybrid system.

Figure. Hybrid-Search-Process-Flow

Data Storage Architecture

One Dataset, Two Indexes:

You maintain a single source of truth but index it twice:

Index 1: Keyword Index (Elasticsearch/BM25)

• Stores: Policy numbers, customer names, transaction history, personal details

• Format: Inverted index (word → documents mapping)

• Update Frequency: Real-time as data changes

Example:

Customer Record:

POL123456 | John Doe | paid $500 | 2024–05–20 | status: pending

Index Entry:

“POL123456”: [customer_doc_1]

“John”: [customer_doc_1]

“paid”: [customer_doc_1]

“pending”: [customer_doc_1]

Index 2: Vector Database (Pinecone/ChromaDB)

• Stores: Policy rules, regulations, FAQs, insurance guidelines

• Format: Dense embeddings (converted via embedding model)

• Update Frequency: When new policies or rules are released

Example:

Document: “Payments typically take 24–48 hours to reflect in your account”

↓ (Embedding Model: OpenAI text-embedding-3-small)

Vector: [0.12, -0.45, 0.89, …, 0.34] (1536 dimensions)

Stored as: {

“id”: “faq_payment_01”,

“text”: “Payments typically take…”,

“vector”: [0.12, -0.45, 0.89, …, 0.34],

“metadata”: {“type”: “faq”, “category”: “payments”}

}

Part 4: Real-World Example — Insurance Customer Support

The Scenario

Customer Query:

“My policy number is POL123456. I paid my instalments last week, but it’s not showing in the system.”

System Goal: Find relevant information and generate a helpful response

Step 1: Query Processing & Parallel Search Execution

The system immediately branches the query to both search engines simultaneously.

Keyword Search Path (BM25)

What Happens:

1. Tokenize query: [“My”, “policy”, “number”, “is”, “POL123456”, “I”, “paid”, “my”, “instalments”, “last”, “week”, “but”, “it’s”, “not”, “showing”, “in”, “the”, “system”]

2. Identify important terms: [“policy”, “number”, “POL123456”, “paid”, “instalments”, “showing”, “system”]

3. Query the inverted index:

Search: policy AND number AND POL123456

Results:

• doc_customer_1 (POL123456 customer record)

• doc_payment_history_1 (POL123456 payment history)

Search: paid AND installment

Results:

• doc_payment_transaction_2 (recent payment)

• doc_payment_guide_3 (how to pay instalments)

4. Apply BM25 scoring:

doc_customer_1: 0.95 (exact match on all key terms)

doc_payment_history_1: 0.92 (matches policy number + payment terms)

doc_payment_transaction_2: 0.87 (matches paid + instalments)

doc_payment_guide_3: 0.78 (matches paid + instalments but generic)

Result: Returns top documents with exact matches on policy number and payment information

Vector Search Path (Embeddings)

What Happens:

1. Convert query to vector via embedding model:

Query: “My policy number is POL123456. I paid my instalments last week, but it’s not showing in the system.”

↓ (Embedding Model)

Query Vector: [0.23, -0.51, 0.87, -0.12, …, 0.45] (1536 dimensions)

2. Search vector database for similar vectors:

Compare Query Vector against all document vectors

Using Cosine Similarity

3. Get similarity scores:

doc_payment_processing_guide: 0.92 (cosine similarity)

• Content: “Payments take 24–48 hours to process”

• Why matched: Intent alignment with “not showing”

doc_payment_delay_faq: 0.88

• Content: “Why isn’t my payment reflected yet?”

• Why matched: Similar phrasing and intent

doc_system_updates: 0.76

• Content: “System updates may delay transactions”

• Why matched: Related to “not showing in system”

doc_transaction_issues: 0.71

• Content: “Troubleshooting payment problems”

Result: Returns semantically similar documents even with different wording

Step 2: Combining & Ranking Results

This is where the magic happens. Results from both searches are merged and re-ranked.

Reciprocal Rank Fusion Algorithm:

BM25 results are ranked 1–10, Vector results are ranked 1–10. We convert ranks to scores:

Score = 1 / (rank + 1)

Calculation:

From Keyword Search:

1. doc_customer_1: Rank 1 → Score = 1/2 = 0.500

2. doc_payment_history_1: Rank 2 → Score = 1/3 = 0.333

3. doc_payment_transaction_2: Rank 3 → Score = 1/4 = 0.250

From Vector Search:

1. doc_payment_processing_guide: Rank 1 → Score = 1/2 = 0.500

2. doc_payment_delay_faq: Rank 2 → Score = 1/3 = 0.333

3. doc_system_updates: Rank 3 → Score = 1/4 = 0.250

Combined Scores:

doc_customer_1: 0.500 + 0 = 0.500

doc_payment_processing_guide: 0 + 0.500 = 0.500

doc_payment_delay_faq: 0 + 0.333 = 0.333

doc_payment_history_1: 0.333 + 0 = 0.333

doc_payment_transaction_2: 0.250 + 0 = 0.250

Final Ranking (Top 5):

1. doc_customer_1 (Score: 0.500)

• Exact policy number + payment info

2. doc_payment_processing_guide (Score: 0.500)

• “Payments typically take 24–48 hours to appear”

3. doc_payment_delay_faq (Score: 0.333)

• “Why is my payment not showing yet?”

4. doc_payment_history_1 (Score: 0.333)

• Complete transaction history for POL123456

5. doc_payment_transaction_2 (Score: 0.250)

• Recent payment confirmation details

Step 3: Context Preparation for LLM

The top 5 ranked documents are compiled as context:

CONTEXT:

Document 1 (Customer Record):

Policy: POL123456

Customer: John Doe

Last Payment: $500 on 2024–05–20

Status: Processing

Account Balance: $0 (paid)

Document 2 (Payment Processing Guide):

All payments to our system are processed within 24–48 business hours.

Once processed, they will be reflected in your account immediately.

Processing may take longer during weekends or holidays.

Document 3 (Payment Delay FAQ):

Q: Why is my payment not showing yet?

A: Common reasons include:

• Payment is still processing (24–48 hours)

• Payment was made on weekend/holiday

• Browser cache needs clearing

Document 4 (Transaction History):

2024–05–20: Payment received — $500

2024–05–19: Due amount — $500

Status: Pending reflection (normal processing)

Document 5 (Payment Confirmation):

Confirmation ID: CONF_12345

Amount: $500

Received: 2024–05–20 14:30 UTC

Expected Reflection: 2024–05–21 or 2024–05–22

Step 4: LLM Generation

The context and original query are sent to a Large Language Model:

Prompt Sent to LLM:

Context:

[Top 5 documents above]

User Query:

“My policy number is POL123456. I paid my installment last week,

but it’s not showing in the system.”

Task: Based on the context above, provide a helpful response to the customer.

LLM Generated Response:

Thank you for contacting us! I found your account for policy POL123456.I can see that your payment of $500 was successfully received on May 20th, 2024. This is completely normal. According to our payment processing guidelines, all payments take 24–48 business hours to be reflected in your account system. Since your payment was received on May 20th (Wednesday), you should see it updated in your account by May 21st or 22nd at the latest. If it hasn’t appeared by May 22nd evening, please reach out to us immediately so we can investigate further.

Your account confirmation number is CONF_12345 for your records.

Is there anything else I can help you with?

Why Hybrid Search Wins a Comparison:

Scenario A: Customer Asks About Payment Issues

Scenario B: Customer Searches for Exact Policy Number

Implementation Considerations

Stack Choices

For Keyword Search:

• Elasticsearch (most popular)

• Apache Solr

• MeiliSearch (modern, user-friendly)

For Vector Search:

• Pinecone (managed, easy to use)

• Weaviate (open-source, flexible)

• ChromaDB (lightweight, for small projects)

• Milvus (high-performance, scalable)

For Embedding Models:

• OpenAI text-embedding-3-small (best quality)

• OpenAI text-embedding-3-large (better but slower)

• HuggingFace (open-source options)

• Cohere Embed (good quality)

For LLM:

• GPT-4o (best quality)

• Claude 3.5 Sonnet (excellent reasoning)

• Llama 2/3 (open-source, self-hosted)

• Mistral (lightweight, fast)

Data Update Strategy

For Keyword Index:

• Real-time updates recommended

• Event-based: trigger updates on data changes

• Example: Payment received → immediately update keyword index

For Vector Database:

• Batch updates (daily/weekly)

• Triggered updates for critical docs

• Example: New insurance policy → re-embed and add to vector DB

Key Takeaways

1. Hybrid search isn’t choosing between keyword and vector search — it’s leveraging both.

2. BM25 (keyword search) excels at exact matches, perfect for:

• Policy/account numbers

• Customer names

• Structured data

• Speed-critical queries

3. Vector search excels at understanding intent, perfect for:

• Natural language questions

• Semantic matching

• Contextual understanding

• Complex queries

4. Together, they create a system that is:

• Fast (keyword search handles exact matches instantly)

• Smart (vector search catches intent)

• Accurate (ranking combines both strengths)

• Scalable (can handle enterprise workloads)

5. Use cases that benefit most:

• Customer support systems

• Internal knowledge bases

• E-commerce search

• Legal/compliance document search

• Healthcare information retrieval

• Insurance claim processing

Conclusion

Hybrid search is already being adopted across enterprise search, e-commerce, and customer support platforms — anywhere that both precision and semantic understanding matter.

The beauty of hybrid search is that you don’t have to choose. You get precision from keyword search and intelligence from vector search. Combined with modern LLMs, you can build customer support systems that understand intent, find exact information, and generate helpful responses automatically.

Whether you’re building a RAG system, improving customer support, or creating an intelligent knowledge base, hybrid search should be in your toolkit.

Ready to implement? Start with a simple setup using Elasticsearch for keyword search and Pinecone for vector search. You’ll see immediate improvements in search quality.

Further Reading

• Understanding BM25 Algorithm
• Vector Databases Explained
• Reciprocal Rank Fusion
• RAG Systems and LLMs

Have you implemented hybrid search in your projects? Share your experience in the comments below!

AI #VectorSearch #KeywordSearch #HybridSearch #RAG #MachineLearning #SearchEngineering #ElasticSearch #Pinecone #DataRetrieval