HyDE: Search With the Answer You Wish You Had

Why a question is a bad search query

Standard RAG embeds the user’s question and asks FAISS to find similar things. That should work — and most of the time it does. But there’s a subtle asymmetry in what the embedding model is doing that causes the failures it does have.

A user asks:

“What causes inflation?”

What this query embedding represents, in vector space, is the question itself — a short, interrogative, vocabulary-light string. The embedding model maps it to a region of the space populated by other questions: things shaped like “what causes X”, “why does Y happen”, etc.

Meanwhile, the document chunks in your index look like this:

“Inflation arises from a combination of demand-pull factors (excess aggregate demand outpacing productive capacity), cost-push factors (rising input costs propagating through supply chains), monetary expansion, and inflationary expectations becoming self-fulfilling…”

The chunk lives in a different region of vector space — populated by other declarative statements, things shaped like long expository explanations.

The query and the chunk are about the same thing, but they’re shaped differently. The embedding distance between them is bigger than the conceptual distance suggests. Vector search has to bridge that gap, and sometimes it doesn’t fully succeed.

HyDE — Hypothetical Document Embeddings — has a clever solution. Instead of searching with the question, generate a fake answer first, embed that, and search with the fake answer’s embedding. The fake answer lives in the same vector-space region as real document chunks. Its embedding lands much closer to the actually-relevant chunks than the query embedding ever could.

You’re searching with a document-shaped probe. The shape matches what you’re looking for. Retrieval gets cleaner.

The picture

STANDARD RAG:

  Query                    Document chunks
  "What causes             "Inflation arises from
   inflation?"              demand-pull factors..."
       │                          │
       ▼                          ▼
  ┌─────────┐              ┌──────────────┐
  │ embed   │              │ embed        │
  └─────────┘              └──────────────┘
       │                          │
       ▼                          ▼
  query vector              chunk vectors
  (lives in                 (live in
   "questions" region)      "documents" region)
       │                          │
       └──────cosine sim?─────────┘
              (worked across regions — gap)

HyDE:

  Query                                          
  "What causes inflation?"                       
       │                                         
       ▼                                         
  ┌─────────────────────────────────────┐        
  │ LLM: write a fake "answer doc"      │        
  │ → "Inflation arises from increased   │       
  │    money supply, supply shocks..."   │       
  └─────────────────────────────────────┘        
       │                                         
       ▼                                         
  ┌─────────┐                Document chunks     
  │ embed   │                 (same as before)   
  └─────────┘                       │            
       │                            ▼            
       ▼                        chunk vectors    
  hypothetical doc vector        (documents     
  (lives in "documents"           region)       
   region — same as chunks!)         │           
       │                             │           
       └─────cosine sim───────────────┘           
            (now both in same region — clean signal)

The embedding alignment is the entire trick. By the time both sides of the cosine similarity calculation are document-shaped strings, the model’s natural geometry works in your favor rather than against it.

How the pipeline runs

Two crucial things to notice:

1. The hypothetical document is fake. The LLM generates it without seeing any of your indexed documents. It’s just predicting “what would a plausible answer to this question look like?” The hypothetical might be factually wrong, vague, or even slightly misleading. That’s fine — it’s not the answer; it’s just a search probe.

2. The hypothetical is thrown away. Once we use its embedding to find the right real chunks, we discard it. The answer LLM never sees the hypothetical. It only sees the actual retrieved document chunks. The hypothetical’s job ends the moment we have its vector.

The hypothetical exists for one purpose: to be a better-shaped search query than the user’s original question. After it does that job, it’s done.

Generating the hypothetical

The reference prompt is direct:

messages = [
    {"role": "system", "content":
        "You are an expert at generating detailed, in-depth documents "
        "that directly answer questions. Generate a document that would "
        "be found in a knowledge base as the perfect answer."},
    {"role": "user", "content":
        f"Given the question '{query}', generate a hypothetical document "
        f"that directly answers this question. The document should be "
        f"detailed and in-depth. The document size should be exactly "
        f"{chunk_size} characters."},
]
hypothetical = llm.chat(messages)

A few things to notice in the prompt:

• “Detailed and in-depth” — explicitly asks for document-shaped prose, not a short answer. A one-line answer would have a query-like embedding and defeat the purpose.
• “The document should be exactly chunk_size characters" — matches the size of the chunks in the index. Embeddings are a little sensitive to length; a hypothetical that matches your chunk size embeds in a more comparable place. The "exactly" is aspirational — LLMs don't hit exact character counts — but the model does land in the right ballpark.
• **temperature=0.7* — notably higher than the typical 0.0 used for extraction tasks. The hypothetical isn't being extracted; it's being imagined*. A bit of variability is actually good — different runs produce slightly different probes, which can be combined or used to explore the embedding space more broadly.

The reference implementation uses gpt-4o (not mini) for hypothetical generation. The reason: the better the LLM, the more domain-appropriate the fake document, and the closer its embedding lands to the right region. For specialized domains (medical, legal, financial), a stronger model produces better hypotheticals.

Worked example

Take a query against a corpus of climate-science documents:

“How does ocean acidification affect coral reefs?”

Step 1: generate the hypothetical. The LLM produces something like this (~1,000 characters):

“Ocean acidification, driven by the absorption of atmospheric CO₂ into seawater, fundamentally disrupts the chemical environment in which coral reefs build and maintain their calcium carbonate skeletons. As seawater pH falls and carbonate ion concentrations decline, the saturation state of aragonite — the form of calcium carbonate corals use — drops below thresholds at which calcification can proceed efficiently. Reef-building corals respond with reduced growth rates, increased skeletal fragility, and impaired recovery from physical damage. Combined with the temperature stress that often accompanies acidification (since both stem from rising atmospheric CO₂), the result is widespread coral bleaching, mass mortality events, and the gradual degradation of reef structures across tropical and subtropical waters…”

This may or may not be 100% accurate — the LLM might overstate the role of aragonite saturation, or get a number slightly wrong — but it doesn’t matter for retrieval. What matters is that this text is shaped exactly like the actual scientific chunks in the index: same vocabulary (acidification, carbonate, aragonite, calcification), same prose register (formal, declarative, multi-sentence), same length.

Step 2: embed the hypothetical and search. The hypothetical’s embedding lands deep in the “scientific prose about coral acidification” neighborhood of vector space. FAISS finds real chunks living in that same neighborhood:

Each of these is a real, scientifically-cited chunk from the actual corpus. None of them necessarily match the hypothetical’s exact claims — but they all sit in the same embedding region. Because the hypothetical’s embedding is in that region, the real chunks come up easily.

Step 3: discard hypothetical, generate answer. The answer LLM gets the three real chunks above (not the hypothetical) and writes a properly grounded answer, citing the real data: “Aragonite saturation has dropped ~40% since pre-industrial times. Juvenile corals show 25–50% reduced calcification rates…” — accurate, sourced, exactly the kind of answer RAG is supposed to produce.

The hypothetical did its job. It got us into the right neighborhood. Then it stepped aside.

What about the same query without HyDE?

The query “How does ocean acidification affect coral reefs?” embedded as-is would also find these chunks — vector search isn’t broken. But the ranking would be noisier. You might see:

• A few of the right chunks at ranks 1–2
• A chunk titled “Frequently Asked Questions about Ocean Health” (matches “ocean”) at rank 3
• A chunk about climate-related coral migrations (related but tangential) at rank 4
• An aragonite-specific chunk that should be top-3 stuck at rank 6

HyDE doesn’t always change the top-1, but it consistently tightens the ranking — fewer thematically-adjacent chunks, more directly-relevant ones. On retrieval benchmarks, the typical lift in Recall@5 is around 10–20% for query distributions where the gap between query language and document language is large.

A small but interesting wrinkle: factual accuracy doesn’t matter

This is the thing that initially feels weird about HyDE. The hypothetical can be wrong. It can be confidently wrong. It can hallucinate dates, misstate causal mechanisms, invent statistics. The retrieval still works — sometimes better than if the LLM had hedged or refused to commit.

The reason is that vector search isn’t checking facts. It’s measuring embedding similarity. An embedding doesn’t care whether the text is true; it cares about the topical and stylistic shape. A confidently-wrong scientific paragraph has the same shape as a confidently-correct one, and embeds to nearly the same region. Either one finds the right chunks.

This is liberating in two ways:

1. You don’t need to use your most expensive, hallucination-resistant model for the hypothetical step. Mid-tier models work fine.
2. You don’t need to verify the hypothetical before using it. It’s used and discarded in milliseconds.

The retrieved chunks — the real ones — anchor the final answer in reality. They’re factually trustworthy because they’re verbatim from the indexed corpus. The hypothetical is just a probe that helps you find them.

The essence in a few lines

def hyde_retrieve(query):
    # 1. Generate a hypothetical answer document
    hypothetical = llm.chat([
        {"role": "system", "content": "Write a detailed document answering the question."},
        {"role": "user", "content": f"Question: {query}\n\nDocument:"},
    ])

    # 2. Embed the hypothetical (NOT the query)
    hyde_embedding = embedder.embed_text(hypothetical)

    # 3. Search with the hypothetical's embedding
    results = vector_store.search(hyde_embedding, k=3)

    # 4. Return real chunks; hypothetical is discarded
    return [r.document.content for r in results]

# In the full pipeline:
chunks = hyde_retrieve(query)
answer = llm.chat_with_context(query, chunks)

Three lines of “what HyDE actually does” wrapped in a normal RAG pipeline. The technique is small. The shift in perspective — what if we searched with a document-shaped probe instead of a query? — is the whole insight.

Knobs you might turn

The interesting tuning lever is **model_name*. Standard RAG can use gpt-4o-mini everywhere happily. HyDE is one of the few places upgrading to gpt-4o for the retrieval* step (not just the answer step) actually pays off — because the better the model, the more domain-fluent the hypothetical, and the cleaner the embedding alignment.

Other tuning is largely pro-forma. The defaults are well-chosen. If you’re seeing weak retrieval on a specific corpus, the first thing to try is upgrading the hypothetical-generation model rather than fiddling with knobs.

The cost — small but real

HyDE adds one LLM call per query. Standard RAG: 1 LLM call (the answer). HyDE: 2 LLM calls (hypothetical + answer).

With gpt-4o-mini for the hypothetical (~500 input tokens for the prompt + ~250 output tokens for a 1000-char hypothetical):

• Cost per hypothetical: ~$0.0002
• Latency added: ~500ms

With gpt-4o (recommended for HyDE):

• Cost per hypothetical: ~$0.004
• Latency added: ~700ms

For most production RAG systems, doubling LLM calls per query is significant in cost terms but trivial in absolute terms (still well under a cent). The latency hit — half a second to a second — is the bigger consideration. If you have a 2-second SLA on responses, HyDE eats half your budget for what may be a small quality gain. If you have a 5-second budget, it fits easily.

Where this earns its keep, and where it doesn’t

HyDE is most useful when:

• Queries are short and document chunks are long. This is the form-factor mismatch HyDE is built for. A user types five words; documents have 200-word paragraphs. The asymmetry hurts standard retrieval; HyDE’s hypothetical fixes it.
• Your corpus uses specialized vocabulary that users don’t. Medical, legal, scientific, technical. The hypothetical injects domain vocabulary into the search probe even when the user types in plain English.
• You can afford one extra LLM call per query. If latency budget allows ~1 extra second, and cost is fine doubling, HyDE is essentially free upside.

It earns less when:

• Queries are already long and detailed. A 200-word query already has document-like shape. The hypothetical isn’t adding much.
• Latency is hard-capped under 1 second. Generating a hypothetical adds 500ms-1s. If sub-second is mandatory, you can’t afford it.
• The LLM doesn’t know your domain. For very obscure corpora — internal company jargon, niche scientific subfields, proprietary product names — the LLM can’t generate a plausible hypothetical. It’ll guess, and the guess might land in the wrong embedding region. In these cases, Document Augmentation (generating questions at index time) is often a better choice — it pre-builds the bridges from real chunks rather than guessing them at query time.
• You’re already using Document Augmentation. They solve the same problem from opposite sides. Combining them is wasteful and sometimes hurts. Pick one.

HyDE vs Document Augmentation — the same idea, opposite directions

These two techniques are siblings. Both close the query-document linguistic gap. They differ in when they do the work:

The economics determine the right pick. If you have a stable knowledge base that gets queried thousands of times per day, Document Augmentation amortizes its big upfront cost into nearly-free query time. If your corpus changes daily but query volume is modest, HyDE adds modest per-query cost without huge upfront investment.

For the rare middle case where both apply — stable corpus AND high query volume AND budget for both — Document Augmentation usually wins because its retrieval is faster and per-query cost is lower. HyDE is the fallback when index-time augmentation isn’t feasible.

The bigger idea worth taking with you

The interesting move in HyDE isn’t generating a fake answer. It’s searching with the form of the thing you want to find, not the form of what you have.

Standard retrieval treats “the query” as immutable — whatever the user typed is what gets embedded. HyDE recognizes that the user’s typed query is already a translation, often a poor one, of what they actually want. The user wants documents about a topic. They expressed that want as a question. The question is a lossy proxy. So HyDE un-translates — uses an LLM to render the want back into something document-shaped — before searching.

This pattern shows up in surprising places once you see it. Query rewriting does the same thing for keyword queries. Step-back prompting does it for abstract queries. Multi-vector retrieval (storing both questions and chunks) does it from the index side. They’re all instances of the same insight: the natural form of a search query isn’t always the optimal form for finding things, and a small amount of preprocessing can dramatically realign the search probe with the search target.

In other words: the user’s query is a hint about what they want, not a specification. Treat it as a hint. Improve it before sending it down the pipeline. HyDE is the most direct expression of this idea, and one of the cheapest upgrades to standard RAG you can make at query time.

A Final Thought

“Good retrieval isn’t just about better embeddings — it’s about sending the right shape into the vector space.”

Connect with me: LinkedIn | Portfolio