Multi-Turn RAG: Maintain Retrieval Context Across Conversation Turns Without Losing Relevance

Last week we designed citations that users can trust and verify. This week we tackle what happens when users ask follow-up questions, and your RAG system forgets what the conversation was about.

Photo by Igor Omilaev on Unsplash

Single-turn RAG answers questions. Multi-turn RAG has conversations.

Introduction

Most RAG systems are stateless. Each query hits retrieval independently, as if the user just walked in the door. That works for one-shot questions. It falls apart the moment a user asks a follow-up. “What about for contractors?” means nothing without knowing the previous turn was about remote work policy. “Can you go deeper on point 2?” requires remembering what point 2 was.

Multi-turn RAG keeps the conversation coherent by carrying context across turns, without stuffing the entire chat history into every retrieval call or blowing up your token budget.

Why Single-Turn RAG Breaks in Conversations

Turn 1: “What is our parental leave policy?”: Retrieves correct chunks. Good answer.

Turn 2: “How does it differ for part-time employees?”: Retrieves random chunks about part-time schedules because the query has no mention of parental leave.

Turn 3: “And what about in the UK office?”: Retrieves chunks about the UK office generally, missing the parental-leave-for-part-time-in-UK intersection entirely.

Each turn loses more context. By turn 3, the user is frustrated and the system is retrieving irrelevant documents.

The Core Problem: Retrieval Has No Memory

Your LLM sees conversation history in its context window. Your retriever does not. It receives one query at a time and searches your corpus based only on that query. When the query is a fragment like “what about contractors?” the retriever has no idea what “what about” refers to.

The fix is not giving your retriever the full conversation history. The fix is giving it a resolved, self-contained query that captures the real intent of the current turn.

The 4 Techniques for Multi-Turn RAG

1) Query Resolution Use the LLM to rewrite the current turn into a standalone query using conversation history.

User turn: “How does it work for contractors?” Resolved query: “How does the remote work policy apply to contractors?”

This is the single most important technique. We covered query transformation previously, but in multi-turn conversations it becomes essential rather than optional. Every follow-up turn should pass through resolution before hitting retrieval.

2) Conversational State Tracking Maintain a lightweight state object that captures the current topic, entities, and constraints mentioned so far.

conversation_state:
  topic: "remote work policy"
  entities: ["contractors", "UK office"]
  constraints: ["part-time employees"]
  active_doc_refs: ["policy_remote_v3"]
  turn_count: 3

Pass relevant state fields into the query resolution prompt so the LLM has structured context, not just raw chat history. This keeps resolution accurate even when conversation history is long or meandering.

3) Retrieval Carry-Forward When a follow-up question narrows or extends the previous question, carry forward the previous turn’s retrieved chunks as candidate context alongside new retrieval results.

Turn 1 retrieves chunks A, B, C about parental leave.

Turn 2 asks about part-time employees. New retrieval finds chunks D and E. But chunks A and B from turn 1 are still relevant. Carry them forward.

Merge carried chunks with new results, then rerank the combined set against the resolved query. This prevents the system from losing context that was already found.

Set a carry-forward limit. Carry chunks from the last 1 to 2 turns only. Older chunks are usually no longer relevant and add token cost.

4) Conversation Summarization for Long Sessions After 5 to 8 turns, raw conversation history becomes too long to pass into resolution prompts efficiently. Summarize the conversation into a compact state.

Replace the full history with a 3 to 5 bullet summary: topic, key facts established, open questions, and active document references.

Use this summary as input to query resolution instead of the raw turns. This keeps resolution fast and focused regardless of conversation length.

Query Resolution Prompt for Multi-Turn

You are resolving a follow-up question into a standalone
search query for a RAG system.

Conversation state:
- Topic: {topic}
- Entities mentioned: {entities}
- Constraints: {constraints}
- Active documents: {active_doc_refs}

Last 2 turns:
{recent_history}

Current user message: {current_message}

Rewrite the current message as a self-contained search query
that includes all necessary context from the conversation.
Keep it under 25 words. Do not add assumptions.
Return only the rewritten query.

Multi-Turn RAG Pipeline

state = get_conversation_state(session_id)
history = get_recent_turns(session_id, last_n=2)

resolved_query = resolve_query(
    current_message=user_input,
    state=state,
    history=history
)

new_chunks = hybrid_search(resolved_query, filters=metadata_filters, top_k=10)
carried_chunks = get_previous_chunks(session_id, last_turns=1)

merged = deduplicate(new_chunks + carried_chunks)
reranked = rerank(resolved_query, merged, top_k=5)
compressed = compress(resolved_query, reranked, top_k=3)

answer = generate(resolved_query, context=compressed, history=history)

update_conversation_state(session_id, resolved_query, reranked)

What to Store Between Turns

Store the conversation state object: topic, entities, constraints, active document references, and turn count.

Store the top chunk IDs from the current turn for carry-forward.

Store a running summary if the conversation exceeds 5 turns.

Do not store full chunk text between turns. Store IDs and re-fetch on the next turn. This keeps session storage small.

Do not store raw conversation history beyond 3 to 5 turns. Summarize and rotate.

Handling Topic Switches

Not every follow-up is a follow-up. Sometimes the user changes topic entirely.

“Thanks for the parental leave info. Now, what is our expense reimbursement process?”

Your query resolution should detect topic switches and reset the conversation state rather than carrying stale context forward. A simple heuristic works: if the resolved query has low similarity to the current state topic, reset.

Alternatively, add a topic-change detector, a small classifier or even a keyword check, before resolution. On topic switch, clear carried chunks and reset the state.

Metrics to Track

Resolution accuracy: Does the resolved query capture the true intent of the follow-up. Sample 20 resolutions weekly and judge manually.

Retrieval precision across turns: Does precision at 3 hold steady from turn 1 to turn 5, or does it degrade. Degradation signals weak resolution or missing carry-forward.

Carry-forward utility: How often do carried chunks appear in the final reranked top 3. If never, carry-forward is adding cost without value. If frequently, it is essential.

Topic switch detection rate: How often does the system correctly detect a topic change versus incorrectly carrying stale context.

Token budget per turn: Total tokens in context should not grow linearly with conversation length. If it does, your summarization is not working.

Common Pitfalls and Quick Fixes

Passing full chat history to retrieval. Fix by always resolving to a standalone query. Raw history is noisy and misleads search.

No carry-forward, so follow-ups lose prior context. Fix by carrying the top 3 chunk IDs from the previous turn and merging before reranking.

Conversation state grows forever. Fix by summarizing after 5 turns and resetting carried chunks older than 2 turns.

Topic switches pollute retrieval with stale context. Fix by detecting topic changes and resetting state.

Resolution prompt is too loose and adds assumptions. Fix by keeping the prompt strict: “Do not add assumptions. Use only information from the conversation.”

Carried chunks dominate new retrieval. Fix by reranking the merged set against the resolved query so fresh, more relevant chunks can outrank carried ones.

Try It Now

Add query resolution to one multi-turn RAG flow using the last 2 turns and a conversation state object.

Implement chunk carry-forward from the previous turn and merge with new retrieval results.

Test with a 5-turn conversation and check whether retrieval precision holds across turns.

Add a topic switch detector that resets state when the subject changes.

Track token budget per turn and verify it stays flat as conversations grow.

Conclusion

Single-turn RAG answers isolated questions. Multi-turn RAG has real conversations. The key is resolving every follow-up into a standalone query, carrying forward relevant context from previous turns, and keeping conversation state compact. When your retriever understands the full intent behind “what about contractors?”, without needing the user to repeat everything, the experience transforms from a search box into a trusted advisor.