FAQ Extraction is a System Design Problem, Not a Prompting Problem

Why we steered our strategy from “Summarization” to “Knowledge Governance.”

I distinctly remember the moment I realized our standard RAG roadmap for FAQ extraction wasn’t going to work.

I was at Tripadvisor (Viator), reviewing the results of a pilot we were running to reduce customer service calls and user drop off at checkout. Our users did not have all the information they needed. My team had built a standard pipeline, retrieve relevant chat logs, cluster them, and summarize an answer. It was the industry standard approach. Then we saw this generated FAQ pair for a popular hiking tour:

Q: Is food included? A: Yes, pets are welcome on this trail.

Legacy FAQ Systems

We dug into the source logs. A customer had asked, “Are dogs allowed?” and the agent confirmed they were. The LLM had correctly retrieved the sentiment (“Yes, allowed”) but completely hallucinated a new question (“Is food included?”) to fit the positive vibe. We knew that if we shipped this. We would pay for that mistake in refunds, negative reviews, and costly support tickets.

Another challenge was in e-commerce, a t-shirt’s specs stay static. But travel inventory is volatile. A pool might be sparkling in photos but closed for renovation today. A “breakfast included” tag can flip overnight due to a supplier change.

This made us realize, we didn’t have a prompting problem. We had a system design problem.

The Diagnosis: Why We Were Failing

My team was burning cycles trying to “prompt engineer” their way out of hallucinations. We conducted a deep-dive into our source data to find the root cause. The findings were stark.

Roughly 30% of customer support conversations that mentioned a specific product weren’t even opened on that product’s page. In travel, customers are chaotic browsers. A user might be looking at a “Skip-the-Line Eiffel Tower” tour but chatting with support about a “Louvre Museum” ticket they booked yesterday.

Because our pipeline blindly clustered questions based on chat logs, we were feeding the LLM contaminated data. The model wasn’t hallucinating; it was faithfully summarizing noise.

Why Legacy Systems or naive RAG fail?

If we continued down the path of “better topic modeling,” we would just build a faster engine for wrong answers. We parked the linear “cluster-and-summarize” roadmap and reframed the problem as knowledge system design with design targets of 90% hallucination reduction and 30% fewer informational support tickets for covered products. These were our north-star metrics, not yet validated outcomes but they shaped every architectural decision that followed.

Thereafter the question wasn’t “how do we generate better FAQs?” It’s “how do we build a system that keeps FAQs true over time, at scale?”

A Better Mental Model: FAQs as a Knowledge Lifecycle

A production FAQ system must do five things:

1. Discover latent customer questions — not just the ones people ask, but the ones they would ask
2. Ground every answer in verifiable evidence
3. Maintain consistency across related facts
4. Detect and correct staleness over time
5. Scale economically across tens of thousands of products

That list pushes you toward three architectural pillars, Self-Adaptive RAG, GraphRAG, and Agentic Validation. They aren’t independent tools, they interlock. RAG provides evidence retrieval, GraphRAG enforces consistency, and agentic governance keeps the system honest as the world changes underneath it.

Modern FAQ Architecture

In the next section, I will walk you through the architecture I would propose in 2026!

1. Self-Adaptive RAG: Optimizing for Answer Stability, Not Similarity

The Strategic Insight: Retrieval Quality is a Variable, Not a Constant.

Standard RAG assumes the user asks a clear question. In my experience at Tripadvisor (Viator), that’s a dangerous assumption. Customers don’t ask “FAQ-shaped” questions in chat logs. Their intent is implicit, fragmented, and often buried in frustration.

The “Dog/Food” hallucination I witnessed during our pilot proved that a single-pass retrieval system was too brittle for production. It trusted the vector search blindly. If the embeddings said “pets” and “food” were semantically close, the model hallucinated a connection.

Solution: Replace this “blind trust” with a Self-Adaptive RAG pipeline.

Drawing on the Self-RAG framework (Asai et al., ICLR 2023), define a new system requirement: the model must be able to critique its own retrieval before generating an answer. I propose a five-stage architecture to treat retrieval configuration as a variable to optimize, not a constant to tune once.

Self Adaptive RAG Pipeline

The Proposed Architecture

1. Latent Question Generation: A system to abstract over raw interactions first. Instead of using a raw chat log, an LLM proposes a candidate question.
2. Initial Retrieval: Hybrid retrieval (semantic + lexical) pulls broad coverage from chat logs, policies, and metadata, but with explicit source authority weighting. Policies and supplier T&Cs are the gold standard; chat logs are useful for phrasing and intent, but never override official documentation.
3. Answer Synthesis: The system generates a draft answer.
4. Answer-Driven Self-Reflection (The “Critic”): This was the core of our proposal. A separate model evaluates the draft for faithfulness and contradictions. If quality is low, the system triggers a retry loop, expanding the search scope (e.g., checking destination-level policies) or switching filters (prioritizing recency over relevance).
5. Learning the Strategy: Successful retrieval paths get cached. The system learns which strategies work for which question types.

Source Authority: The Hierarchy of Evidence

One critical lesson: not all sources are equal.

Chat logs are “hearsay.” If a previous human agent gave the wrong answer (“Yes, breakfast is free”), and your RAG retrieves that chat log, the Critic model might validate the new answer as “faithful” because it matches the (incorrect) source text.

Implement explicit source weighting in the retrieval layer:

[embed]

In the knowledge graph, validated_by edges always point to Gold or Silver sources. If a chat-log-derived answer can't be validated against official documentation, it gets flagged for human review.

The Pseudocode

For engineers, here’s the core loop:

async def run_faq_generation_job(product_id: str, chat_logs: list, policies: list):
    """
    Runs OFFLINE to hydrate the FAQ Knowledge Base.
    Not a real-time user query handler.
    """

    # 1. Intent Extraction (abstract over noisy chat logs)
    # question_type is determined by an LLM classifier: 
    # "cancellation", "accessibility", "inclusions", "logistics", etc.
    latent_questions = await llm.extract_intents(chat_logs)

    for question in latent_questions:
        strategy = get_cached_strategy(question.type) or default_strategy()

        # 2. Hybrid Retrieval with Source Authority Weighting
        # Policies (Gold) > Supplier APIs (Silver) > Chat Logs (Bronze)
        evidence = search_index.retrieve(
            query=question,
            sources=["policies", "supplier_api", "chat_logs"],
            source_priority=["policies", "supplier_api", "chat_logs"],
            strategy=strategy
        )

        # 3. Draft & Critique Loop
        draft = llm.synthesize(question, evidence)
        critique = llm_critic.evaluate(
            draft, 
            evidence,
            checks=["faithfulness", "completeness", "no_contradictions", "source_authority"]
        )

        # 4. Self-Correction (The "Adaptive" part)
        if critique.score < CONFIDENCE_THRESHOLD:
            # Expand scope: destination-level policies, similar products
            expanded_evidence = expand_search_scope(product_id, question)
            draft = llm.synthesize(question, expanded_evidence)
            critique = llm_critic.evaluate(draft, expanded_evidence)
            cache_strategy(question.type, strategy.adapted())

        # 5. Publish or Queue for Human Review
        if critique.score >= PUBLISH_THRESHOLD:
            faq_cache.publish(product_id, question, draft, sources=critique.sources)
        else:
            human_queue.add(product_id, question, draft, reason=critique.failure_reasons)

The key insight: source_priority ensures that a chat log claiming "breakfast is free" gets overridden by a policy document stating "meals not included."

Why Choose This Architecture

Specifically to catch the “Silent Failures” that plague travel data. In the “Dog/Food” example, a Self-Adaptive system would have flagged the draft answer because the retrieved evidence (“dogs allowed”) did not support the generated question (“food included”). The Critic model would have rejected the pair, saving us from a potential refund.

We aimed to target a ~90% reduction in hallucination rates, shifting our focus from “fixing prompts” to “fixing the retrieval logic.”

Handling the Cold Start Problem

The architecture assumes rich chat history but what about new products with zero conversations? Or long-tail tours that get three support tickets a year?

We identified three bootstrapping strategies for sparse-data scenarios:

Metadata-First Generation. For new products, the system generates candidate FAQs directly from structured fields, cancellation policies, included amenities, accessibility tags, duration, and supplier terms. These aren’t derived from customer questions they anticipate them based on what similar products get asked.

Why use an LLM if you have structured data? Consistency of voice. A raw database field like cancellation_policy: FREE_24H becomes "You can cancel for free up to 24 hours before your experience." The UI feels conversational, not like a data dump.

Sibling Product Transfer. A new “Skip-the-Line Colosseum” tour from an established supplier inherits FAQ templates from that supplier’s other Rome products. The knowledge graph makes this trivial: query all products sharing the same operated_by edge, retrieve their FAQ subgraphs, and adapt. The Critic model validates that transferred answers still apply.

*Prerequisite*: This requires robust entity resolution**. In travel data, “Roma Tours”, “Roma Tours Inc.”, and “Roma Tours S.r.l.” often appear as different entities in different systems. If your knowledge graph doesn’t deduplicate suppliers and locations, sibling transfer fails silently. We invested significant effort in entity resolution before GraphRAG became useful.

Confidence-Gated Display. FAQs generated from sparse evidence get a lower confidence score. Below a threshold, they’re either hidden from the product page or displayed with softer language: “Based on similar experiences…” rather than definitive claims. As chat volume grows, the system re-evaluates and promotes high-confidence answers.

The cold-start problem never fully disappears but these strategies meant we could launch with 80% product coverage on day one, rather than waiting months for sufficient chat volume.

2. GraphRAG: Modeling FAQ Knowledge, Not Text

Vector search is excellent at finding similar text. It’s terrible at preserving truth relationships.

FAQs are rarely independent. They share constraints, prerequisites, and policies. Consider three questions about a Vatican tour:

• “Can I cancel my booking?”
• “How long until I get my refund?”
• “What if the tour is cancelled due to weather?”

These questions are deeply interdependent. The refund timeline depends on cancellation eligibility. The weather policy supersedes the standard cancellation policy, full refund, no penalty, regardless of timing.

Generate these FAQs independently and contradictions creep in: one says “free cancellation up to 24 hours,” another mentions “5–7 business days for refunds,” and a third states “no refunds for cancellations.”

This isn’t a hallucination problem. It’s a knowledge representation problem.

Why GraphRAG Works

Microsoft Research’s GraphRAG (Edge et al., 2024) demonstrated that graph-based retrieval dramatically outperforms vector search for questions requiring multi-hop reasoning — hopping from Tour → Supplier → Policy → Exception. The same principle applies to FAQ consistency.

Instead of storing FAQs as flat Q&A pairs, represent them as a knowledge graph:

Nodes represent entities: Products (Skip the Line: Vatican & Sistine Chapel), Suppliers (Roma Tours Inc.), Policies (Cancellation, Refund, Weather), Conditions (24hr notice, Full refund), and Evidence Sources (Supplier T&C, Viator Policy).

Edges capture relationships: applies_to links policies to products, depends_on captures prerequisites, supersedes handles policy exceptions, validated_by traces answers to sources.

At generation time, the system retrieves a subgraph, not text chunks. When a customer asks about cancellation, the LLM receives the full context: the tour, its supplier’s terms, all applicable policies, their dependencies, and which conditions apply.

What This Solves

Consider: Roma Tours updates their weather policy to offer rebooking instead of automatic refunds. With vector search, the “weather cancellation” FAQ might update while the “refund timeline” FAQ still references the old policy — a contradiction that surfaces when a customer tries to rebook during a rainstorm.

With GraphRAG, the supersedes edge propagates the change. Every dependent FAQ gets flagged for regeneration.

This reduces three error classes:

• Inconsistent answers. FAQs sharing policy nodes can’t contradict each other.
• Duplicate FAQs. Fifty Vatican tours from the same supplier link to one cancellation policy. Update once, propagate everywhere.
• Silent policy conflicts. When a supplier changes their refund window, the graph surfaces affected FAQs before customers do.

Knowledge SubGraph

3. Agentic Validation: FAQs That Fix Themselves

Static FAQs decay. Suppliers update policies, prices shift, schedules rotate seasonally — but batch refresh cycles can’t keep up.

Consider the timeline of a typical failure:

• Day 1: FAQ generated: “Free cancellation up to 24 hours before your experience.”
• Day 15: Supplier quietly updates to 48-hour notice. Your FAQ doesn’t know.
• Days 15–45: Customers get wrong information.
• Day 45: Complaint. Manual investigation. Manual fix.

This is the default state of most FAQ systems. Accurate on launch day, degrading from there.

The Fix: Continuous Agentic Validation

Instead of waiting for complaints, deploy agents that continuously validate FAQs against their sources. Singh et al.’s survey on Agentic RAG (2025) provides a comprehensive taxonomy of these systems — agents that dynamically manage retrieval and adapt to changing contexts.

Four agents work in concert:

Monitor Agent. Watches for drift signals: supplier API changes, price feed updates, T&C edits, support ticket spikes, review sentiment shifts. When a Colosseum tour’s price changes from €45 to €52, the Monitor flags every FAQ referencing that pricing.

Validate Agent. Checks flagged FAQs against current sources. Computes confidence scores. If confidence drops below threshold, marks the FAQ stale.

Correct Agent. Regenerates stale FAQs through the Self-Adaptive RAG pipeline, updates knowledge graph edges.

Govern Agent. Enforces quality gates. Price updates auto-publish. Policy changes queue for review. Safety-related changes wait for human approval.

When Agents Fail

Agentic systems introduce new failure modes. We designed for three:

False Positive Floods. If the Monitor Agent fires on every minor price fluctuation, the system drowns in unnecessary regeneration cycles. We implemented a dampening layer: changes below a materiality threshold (e.g., <5% price shift, <12-hour schedule change) get logged but don’t trigger validation unless they persist for 48 hours or co-occur with support ticket spikes.

Regression on Correction. Sometimes the Correct Agent regenerates a worse FAQ than the stale one it replaced — particularly when source data is ambiguous. We added a regression check: before publishing, the system compares the new FAQ’s Critic score against the previous version. If the new score is lower, it queues for human review rather than auto-publishing a degradation.

Confidence Threshold Miscalibration. Set the threshold too high, and valid FAQs get stuck in review queues. Too low, and garbage ships. We treat the threshold as a tunable parameter per FAQ category: pricing FAQs (high cost of error) use a stricter threshold than general amenity FAQs. The Govern Agent logs all decisions, enabling weekly threshold audits based on downstream complaint rates.

No agent system is autonomous forever. The goal is reducing human intervention to the cases that genuinely need judgment — not eliminating it entirely.

In Practice

A price changes in the supplier feed. Within minutes:

1. Monitor detects the delta
2. Validate finds the FAQ stale (confidence: 0.3)
3. Correct regenerates: “Prices start from €52”
4. Govern auto-publishes (low risk)

No support tickets. No complaints. No manual intervention.

Offline Generation, Online Serving

A common question: when does all this computation actually happen?

The Self-Adaptive RAG pipeline doesn’t run on every page view. FAQs are pre-computed artifacts, stored in a cache layer, and served statically. The Agentic Validation loop monitors for staleness and triggers regeneration only when needed, typically a few percent of FAQs per day, not the entire corpus.

[embed]

This separation is what makes the architecture economically viable.

The Economics (Back-of-Envelope)

A quick sanity check on whether this architecture pays for itself:

Costs:

• FAQ generation: ~$0.15 per FAQ (4 LLM calls × ~1K tokens each)
• Agentic monitoring: ~$500/month for streaming infrastructure + spot validation
• Graph maintenance: ~$200/month for hosted graph DB at 50K products

Benefits:

• Average support ticket cost: $5–15 (agent time + platform fees)
• If 10% of tickets are “answered by FAQ if FAQ existed”: 5,000 deflected/month
• Savings: $25,000–75,000/month

Break-even: System pays for itself if it deflects ~200 tickets/month. At Viator scale, that’s a rounding error.

The 90% hallucination reduction target isn’t about cost savings, it’s about preventing the negative value of wrong answers. One viral “the FAQ lied to me” tweet costs more than a year of compute.

Putting It Together

Self-Adaptive RAG discovers questions and generates grounded responses. GraphRAG links related facts and enforces consistency. Agentic Validation monitors for drift and triggers corrections — which flow back through RAG for regeneration.

The loop closes. The system learns. FAQs stay fresh.

Where to Start

You don’t need all three pillars at once:

Phase 1: Self-Adaptive RAG. Replace single-pass RAG with a self-reflective pipeline. Add source authority weighting. This alone reduces hallucinations and improves long-tail coverage. Ship value in weeks.

Phase 2: GraphRAG. Layer in knowledge graph structure. Start with entity resolution for suppliers and locations. Model the relationships that cause the most contradictions (policies, dependencies). Expand as you learn.

Phase 3: Agentic Validation. Add continuous monitoring. Start with high-signal sources: price feeds, supplier APIs. Implement dampening to avoid alert fatigue. Expand coverage as you build confidence.

Each phase delivers standalone value. Each phase makes the next more effective.

The Core Shift

Legacy systems treat FAQs as documents to generate and forget. Modern systems treat them as derived knowledge — views over a living knowledge base that must stay synchronized with reality.

The question isn’t “how do we generate better FAQs?” It’s “how do we build a system that keeps FAQs true over time, at scale?”

Documents decay. Knowledge systems adapt. Build the system.

What’s Next

This architecture is a proposal, not a shipped product. The real test comes when it meets production traffic, edge cases we haven’t imagined, and suppliers who update their terms via fax.

If you’re building something similar, or have solved these problems differently, I’d genuinely like to hear about it. What’s worked? What’s failed spectacularly? The travel-tech community is small enough that we don’t need to repeat each other’s mistakes.

Find me on LinkedIn or drop a comment below.

Further Reading

• Self-RAG (Asai et al., ICLR 2024) — Self-reflective retrieval where the model learns when to retrieve and critiques its own generations. arxiv.org/abs/2310.11511
• GraphRAG (Edge et al., Microsoft Research, 2024) — Graph-based retrieval for questions requiring synthesis across documents. arxiv.org/abs/2404.16130 | github.com/microsoft/graphrag
• Agentic RAG Survey (Singh et al., 2025) — Comprehensive taxonomy of agent-enhanced retrieval systems. arxiv.org/abs/2501.09136