Chain of Verification: the prompting pattern that makes LLM answers check themselves

A recent viral thread claims that a Meta AI technique can make LLMs dramatically more accurate even with zero examples. The technique is real, it is called Chain of Verification (CoVe), and it comes from research on reducing hallucinations by forcing a model to verify its own draft before it responds.

his article is a practical, implementation focused deep dive: what CoVe is, why it works, how to prompt it, and how to ship it in production without exploding cost or latency.

Before we proceed, let’s stay connected! Please consider following me on Medium, and don’t forget to connect with me on LinkedIn for a regular dose of data science and deep learning insights.” 🚀📊🤖

Why few shot prompting breaks in production

Few shot prompting is great for teaching a format. But it is fragile for factuality because the model still does one pass generation: it writes a confident answer and moves on. If it invents a detail early, later tokens tend to build on that error. The CoVe paper frames this as hallucination: plausible looking but incorrect facts that can persist even in large models.

In production, this shows up as:

1. Lists with subtle wrong items
2. Biographies with wrong dates, places, affiliations
3. Closed book QA where one incorrect span poisons the rest
4. Overconfident answers that do not surface uncertainty

CoVe attacks that failure mode by adding a structured verification phase at inference time. Same model, better process.

What Chain of Verification is

Chain of Verification is a multi stage prompting recipe where the model:

1. Drafts an initial response
2. Plans verification questions that would fact check the draft
3. Answers those verification questions independently
4. Produces a final response revised using the verification results

The key detail is step 3: independence. The paper shows that if the model can see its own draft while answering the checks, it can copy the same hallucination again. So the best variants reduce that leakage by factoring the prompts so verification does not condition on the draft.

What the research actually shows (numbers you can quote)

The thread highlights a big accuracy improvement claim. The paper itself reports strong gains across several benchmarks:

List generation tasks: fewer hallucinated entities

On a Wikidata style list task, CoVe more than doubles precision compared to a few shot baseline (0.17 to 0.36), while the average number of hallucinated entities drops sharply (2.95 to 0.68).

Closed book QA: better F1

On MultiSpanQA in a closed book setting, CoVe improves F1 over few shot (0.39 to 0.48).

Long form biographies: higher factuality score

On biography generation, a CoVe variant (factor plus revise) improves FACTSCORE from 55.9 to 71.4 versus few shot.

Also important: classic Chain of Thought prompting does not help here and sometimes hurts, because “thinking step by step” is not the same as checking facts.

The core prompt pattern

You can implement CoVe with either one call or multiple calls.

Option A: single call CoVe (fastest to integrate)

This is the “joint” approach. It is simpler but more prone to copying errors because the verification is in the same context as the draft. The paper still finds improvements.

Use a structured output schema like this:

You are a careful assistant.

Task:
Answer the user question.

Process:
1) Draft an initial answer.
2) List verification questions that would check each factual claim in your draft.
3) Answer each verification question.
4) Produce a final answer that is consistent with your verification answers.
5) If verification is inconclusive, mark the claim as uncertain or remove it.

Return JSON with keys:
draft
verification_questions
verification_answers
final
uncertainties

Option B: factored CoVe (best quality for high stakes)

This is closer to the strongest results. It uses multiple calls and explicitly prevents the verification step from seeing the draft. The paper motivates this because attending to the baseline can cause repetition of hallucinations.

Call structure:

1. Call 1: draft
2. Call 2: plan verification questions (this call can see the draft)
3. Call 3+: answer each verification question independently (this call must not see the draft)
4. Call final: rewrite answer using only the question plus the verification answers

Production ready implementation in Python (model agnostic)

Below is a clean reference implementation. It works with any chat style LLM API: you just need a call_llm(messages) function.

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable, Dict, List, Any

@dataclass
class CoVeResult:
    draft: str
    verification_questions: List[str]
    verification_answers: List[str]
    final: str

def _as_bullets(items: List[str]) -> str:
    return "\n".join([f"{i+1}. {x}" for i, x in enumerate(items)])

def run_cove(
    question: str,
    call_llm: Callable[[List[Dict[str, str]]], str],
    max_questions: int = 8,
) -> CoVeResult:
    # 1) Draft
    draft = call_llm(
        [
            {"role": "system", "content": "You are a helpful assistant that answers clearly and precisely."},
            {"role": "user", "content": question},
        ]
    )

    # 2) Plan verification questions (can see draft)
    plan_prompt = f"""
Given the user question and the draft answer, write up to {max_questions} verification questions
that would fact check the draft. Each question must be answerable without referencing the draft text.

User question:
{question}

Draft answer:
{draft}

Return the questions as a numbered list. Do not answer them.
""".strip()

    q_text = call_llm(
        [
            {"role": "system", "content": "You are a fact checking planner."},
            {"role": "user", "content": plan_prompt},
        ]
    )

    # Simple parser for numbered lists
    verification_questions: List[str] = []
    for line in q_text.splitlines():
        line = line.strip()
        if not line:
            continue
        if line[0].isdigit():
            parts = line.split(".", 1)
            if len(parts) == 2:
                verification_questions.append(parts[1].strip())
    verification_questions = verification_questions[:max_questions]

    # 3) Answer verification questions independently (must NOT see draft)
    verification_answers: List[str] = []
    for q in verification_questions:
        a = call_llm(
            [
                {"role": "system", "content": "Answer the question with the best factual answer you can. If unsure, say unsure."},
                {"role": "user", "content": q},
            ]
        )
        verification_answers.append(a.strip())

    # 4) Final rewrite using verification answers
    rewrite_prompt = f"""
Rewrite the answer to the user question using ONLY the verification answers as evidence.
If a claim is unsupported or uncertain, remove it or mark it as uncertain.
Keep the final answer readable.

User question:
{question}

Verification questions:
{_as_bullets(verification_questions)}

Verification answers:
{_as_bullets(verification_answers)}

Return only the final answer.
""".strip()

    final = call_llm(
        [
            {"role": "system", "content": "You are a careful editor that writes a final verified answer."},
            {"role": "user", "content": rewrite_prompt},
        ]
    )

    return CoVeResult(
        draft=draft.strip(),
        verification_questions=verification_questions,
        verification_answers=verification_answers,
        final=final.strip(),
    )

Why this implementation matches the paper

1. It separates verification execution from the baseline draft, which the paper argues helps avoid repeating hallucinations.
2. It uses verification questions that are simpler than the original generation, aligning with the finding that models answer these checks more accurately than the initial long form answer.
3. It lets you scale cost with max_questions and only pay extra tokens when accuracy matters.

Prompting tips that matter

Prefer “general” verification questions over yes no

The paper compares verification plan styles and finds differences between yes no style and general questions, and notes yes no can be weaker in some setups. In practice, ask “When did X start and end” instead of “Is it true that X happened in 1846.”

Verify claims, not paragraphs

CoVe works best when you force the planner to identify atomic claims. For a biography, that means:

1. Birth date and place
2. Key roles and dates
3. Awards and publications
4. Organization affiliations

Use CoVe selectively

CoVe adds calls and tokens. Use it for:

1. External facing factual content
2. Lists of entities, dates, places
3. Summaries that will be used downstream
4. Safety sensitive explanations where correctness matters

For casual chat, standard prompting is usually enough.

Extending CoVe with tools (RAG plus verification)

The original paper does not use tool use in the verification step, but it explicitly calls out retrieval augmentation as a natural extension.

A strong production pattern is:

1. Draft with citations from retrieval
2. Plan verification questions
3. For each verification question, retrieve again with a focused query
4. Answer using retrieved snippets
5. Rewrite final answer with citations

This often beats plain RAG because it creates targeted retrieval queries per claim instead of one broad retrieval for the entire question.

A simple way to evaluate CoVe in your own product

Set up an offline evaluation with:

1. A dataset of user questions
2. A trusted reference source or labeler

Metrics:

Factual precision (how many claims are correct)

Hallucination rate (incorrect claims per answer)

Coverage (how much useful correct information remains)

Cost and latency

The paper reports both correctness and how many facts were produced, which is important: you want fewer hallucinations without collapsing the answer into something tiny.

Closing thoughts

CoVe is not magic. It is an engineering pattern: turn one generation into a small pipeline where the model has to interrogate itself and then edit. The reason it works is simple: models can often answer narrow verification questions more reliably than they can produce a long, perfectly factual narrative in one shot.