Stream RAG: Building Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage

1. From conventional language models to real-time spoken intelligence

I began this work from the simplest possible abstraction of a language model: a user submits an input query, the model processes it, and the system returns an output. This paradigm is adequate for many text-based interactions because users generally tolerate a short pause while reading or typing. The model receives a complete, semantically coherent request before it must act.

However, this abstraction conceals two fundamental weaknesses. First, the model may not possess the latest or most specialized information required to answer the question. Second, even when the model appears confident, it may generate statements unsupported by reliable evidence. In ordinary text interfaces, users can reread an answer, inspect citations, and identify contradictions. In spoken dialogue, verification is substantially harder because information disappears temporally as it is heard.

Therefore, although the input–model–output architecture provides a useful conceptual baseline, it is not sufficient for trustworthy, natural, real-time voice interaction.

2. Retrieval-augmented generation as an external knowledge mechanism

Retrieval-augmented generation, or RAG, improves this baseline by introducing an external information-retrieval stage. Instead of forcing the language model to answer exclusively from its internal parameters, I allow the system to search a document collection, database, web index, or enterprise knowledge base. The retrieved evidence is then supplied to the language model as additional context.

Conceptually, the process is straightforward:

1. The user submits a query.
2. The retrieval system identifies relevant documents.
3. The language model generates an answer conditioned on those documents.
4. The system may attach citations or source references.

This architecture reduces dependence on memorized knowledge and makes it possible to answer questions concerning recent events, proprietary data, or specialized domains.

Nevertheless, the phrase “no hallucination” should be interpreted as an aspiration rather than a guarantee. RAG can still fail if the query is poorly formulated, if the relevant document is absent, if retrieval ranks an irrelevant passage highly, or if the language model misinterprets the evidence. The real objective is not the absolute elimination of hallucination, which is generally unrealistic, but a measurable reduction in unsupported claims and an increase in evidence-grounded responses.

3. Why spoken dialogue creates a more demanding problem

Voice interfaces intensify the challenge. Consider a user asking a home assistant, “What is the weather?” The assistant should ideally provide the current temperature, atmospheric conditions, expected precipitation, and daily high and low temperatures. The answer must be both factually current and delivered rapidly enough to preserve conversational continuity.

The user may then ask a follow-up question such as, “Should I take an umbrella?” or “Will it still be raining when I leave work?” The system must preserve conversational context while retrieving new evidence when necessary.

Unlike a conventional chatbot, a spoken agent operates under temporal pressure. Human conversation contains interruptions, hesitation, repairs, unfinished sentences, and incremental revelation of intent. A speaker may begin with “What is the weather…” and later append “…near the airport tomorrow morning?” The agent cannot safely assume that the first few words represent the complete request.

My central question was therefore not simply how to add RAG to a voice assistant, but how to integrate retrieval without destroying the natural rhythm of conversation.

4. The target system: a grounded conversational voice agent

The desired architecture is a conversational voice AI agent that receives spoken input and produces a grounded spoken response. It must combine several capabilities:

• speech recognition or direct audio understanding;
• conversational context tracking;
• retrieval and tool invocation;
• evidence-conditioned response generation;
• speech synthesis;
• interruption and turn-taking management.

Accuracy alone is inadequate. A system that eventually produces a correct response after several seconds may still feel unusable. Conversely, a system that responds instantly but fabricates information is equally unacceptable.

I therefore treat voice-agent quality as a multi-objective optimization problem. The system must jointly minimize factual error, retrieval delay, first-token latency, computational cost, and conversational disruption.

Again, the label “no hallucination” represents a design target. In practice, a robust system should additionally expose uncertainty, distinguish retrieved facts from inferences, and refuse to invent an answer when the evidence is insufficient.

5. Why conventional RAG creates an unacceptable latency penalty

A naïve implementation waits until the user finishes speaking, transcribes the complete utterance, formulates a retrieval query, searches the document collection, reranks the results, generates a response, and finally synthesizes audio.

The sequential latency can be approximated as:

[

T_{\text{total}}

T_{\text{endpoint}}

• T_{\text{transcription}}

• T_{\text{query}}

• T_{\text{retrieval}}

• T_{\text{reranking}}

• T_{\text{generation}}

• T_{\text{speech}}. ]

The endpointing term is particularly important. Before beginning retrieval, the system must decide that the user has actually stopped speaking rather than merely pausing. Conservative endpointing reduces interruptions but increases delay.

Even a technically modest delay can feel unnatural. Human listeners interpret prolonged silence as confusion, disengagement, or system failure. A voice agent that pauses for several seconds before every factual answer does not behave like a conversational partner; it behaves like a slow search interface with speech attached.

The conventional pipeline therefore places retrieval entirely on the critical path. Stream RAG attempts to move a substantial portion of that work off the critical path by overlapping retrieval with the user’s ongoing speech.

6. The core insight: retrieve while the user is still speaking

The central idea of Stream RAG is speculative, incremental retrieval. Rather than waiting for the final word, I continuously analyze partial speech and initiate retrieval as soon as the emerging intent becomes sufficiently informative.

Suppose the user says:

“What is the weather today? I am trying to decide whether I should go for a run after lunch.”

The decisive information may already be contained in the first sentence. The second sentence adds motivation but may not materially alter the retrieval target. If the system recognizes this early, it can retrieve the weather data while the user continues speaking.

This creates a temporal overlap:

• the user continues producing speech;
• the system generates or updates a query;
• retrieval executes in parallel;
• retrieved evidence becomes available near the end of the utterance;
• response generation begins with little additional waiting.

The key difficulty is determining when a partial utterance is sufficiently complete. Triggering too early can retrieve information for the wrong intent. Triggering too late forfeits the latency benefit. The problem is therefore an optimal stopping problem under semantic uncertainty.

7. Fixed-interval streaming RAG

The first approach divides the incoming speech into consecutive blocks. Let the complete utterance contain (B) blocks, and let block (b) represent the speech observed up to a particular time.

After receiving each block, the system may generate an intermediate retrieval query:

[ Q_1^T, Q_2^T, \ldots, Q_B^T, ]

where (Q_b^T) is derived from the partial transcript available at block (b).

The associated retrieval jobs can be executed concurrently. This parallelism is important: the system does not necessarily need to wait for one query to finish before initiating the next. As the user’s request becomes more explicit, later queries can refine or replace earlier speculative queries.

For example, the partial utterances might evolve as follows:

• “Who founded…”
• “Who founded Rare Beauty…”
• “Who founded Rare Beauty in 2019?”

The initial query is under-specified. The second identifies the entity. The third contains enough information to retrieve highly relevant documents.

Fixed-interval retrieval is simple and model-agnostic, but it can be computationally wasteful. If every block triggers retrieval, many nearly identical searches may be performed. The architecture must therefore balance temporal granularity against infrastructure cost.

8. Two-stage processing: early retrieval and final response generation

I organize the system into two conceptual stages.

In Stage 1, the agent processes incremental speech blocks and issues speculative RAG queries. Each query produces retrieved evidence (R_b). By the time the user finishes speaking, several candidate retrieval results may already exist.

In Stage 2, the system selects the most useful evidence and generates the final response. The complete user utterance remains available to the voice agent, so speculative retrieval does not force the final answer to rely only on an incomplete transcript.

This separation is essential. Early retrieval is permitted to be provisional; final generation must still consider the complete request. The system can therefore gain latency through speculation without blindly committing to the earliest interpretation.

A reflector component examines intermediate queries or retrieval results and decides which candidate is sufficiently consistent with the final intent. The chosen evidence, denoted (R_b^*), is then supplied to the response generator together with the full speech input.

In effect, Stage 1 attempts to anticipate what information will be required, while Stage 2 verifies that anticipation before producing audio.

9. The reflector as an early-sufficiency estimator

The reflector is the component that decides whether an intermediate query is already sufficient to answer the eventual complete query.

A simplified decision rule can be written as:

[

b^*= \min \left{ b\in[1,B] ;\middle|; \operatorname{reflect} \left( \hat{Q}_b^T,\hat{Q}_B^T \right)

\mathrm{True} \right}. ]

Here:

• (B) is the final speech block;
• (\hat{Q}_b^T) is the query inferred from speech up to block (b);
• (\hat{Q}_B^T) is the query inferred from the complete utterance;
• (b^*) is the earliest block judged sufficient.

This formulation exposes the underlying optimization objective: find the earliest partial interpretation that is functionally equivalent to the final interpretation for retrieval purposes.

Literal textual equivalence is unnecessary. “Who founded Rare Beauty?” and “Who was the founder of the cosmetics company Rare Beauty?” are linguistically different but retrieval-equivalent. The reflector should therefore evaluate informational sufficiency rather than surface similarity.

Possible reflector signals include semantic similarity between queries, overlap among top retrieved documents, stability of entity recognition, agreement between reranker scores, confidence that mandatory query slots have been filled, and estimated probability that future speech will materially alter the retrieval target.

10. Processing speech incrementally from the first acoustic evidence

The streaming process begins as soon as the user speaks. The system receives audio continuously rather than treating the utterance as a single indivisible object.

Depending on the implementation, the conversational model may consume raw audio representations, partial automatic-speech-recognition transcripts, or a combination of both. Partial transcripts are often unstable: a recognizer may revise earlier words after receiving additional acoustic context. Consequently, the retrieval system must tolerate corrections.

For instance, an early transcript might interpret a phrase as “weather in Paris,” while a later block revises it to “weather in Parrish, Florida.” A retrieval request based on the early transcript may be incorrect. Stream RAG must therefore regard early queries as hypotheses rather than immutable commands.

The system also needs to distinguish semantic content from conversational fillers. Expressions such as “um,” “you know,” or “I was wondering whether” usually contribute little retrieval value. A well-designed streaming agent compresses these signals into a concise representation of the emerging information need.

11. Learning when to issue a new retrieval query

Calling RAG at every interval is not always economically or computationally desirable. The second approach therefore trains the agent to decide between two actions:

• NO_QUERY: reuse the previous useful retrieval result;
• NEW_QUERY: formulate and execute a new retrieval request.

The decision depends on the current speech block, earlier blocks, the previous query, and potentially the previously retrieved evidence.

Suppose the user first says, “Who founded Rare Beauty?” The system retrieves documents about Selena Gomez. If the user then adds, “in 2019,” the existing evidence may already contain the founding date, so a new retrieval call may be redundant.

By contrast, if the user continues, “…and what company acquired it later?”, the information need changes materially. A new query becomes necessary.

This mechanism converts retrieval from a fixed periodic operation into a learned tool-use policy. The agent is not merely generating language; it is deciding whether an external action has sufficient expected value to justify its latency and computational cost.

12. Constructing training data with pseudo queries

Training such a decision policy requires supervision at intermediate points in an utterance. Human datasets rarely contain labels specifying the precise moment when retrieval should occur, so I can create supervision through a post-training pipeline.

First, I take progressively longer partial transcripts. For example:

• “Who founded”
• “Who founded Rare Beauty”
• “Who founded Rare Beauty in 2019?”

Second, an LLM or specialized query generator converts each partial transcript into a pseudo retrieval query. Early pseudo queries may be vague, while later ones become more explicit.

Third, each pseudo query is executed against the RAG system. The resulting documents reveal whether the partial query already retrieves evidence useful for answering the complete question.

In the example, even imperfect queries such as “Rare Beauty founder 2019” may retrieve documents stating that Selena Gomez founded or launched Rare Beauty in 2019. If these documents are sufficient, the partial query can be treated as operationally successful.

This method evaluates partial utterances according to downstream utility rather than linguistic elegance. A grammatically awkward query is acceptable if it retrieves the right evidence.

13. Assigning NO_QUERY and NEW_QUERY labels

After generating pseudo queries and retrieving documents, I compare the current retrieval result with the result produced by the previous useful query.

If both queries retrieve substantially the same useful evidence, the current block receives the NO_QUERY label. The model should learn that another tool invocation is unnecessary.

If the new partial query retrieves meaningfully different or more relevant evidence, the block receives the NEW_QUERY label.

This comparison can be based on document overlap, answer-containing passages, retrieval scores, reranker judgments, or an evaluator model that estimates answerability.

Negative sampling strengthens the training procedure. A previous query can deliberately be replaced with an irrelevant or misleading query. The model must then recover by generating the correct query rather than indiscriminately outputting NO_QUERY.

This prevents a degenerate strategy in which the agent avoids retrieval simply to minimize tool calls. The goal is not the fewest possible queries; it is the fewest queries consistent with reliable evidence acquisition.

A production-grade policy should ideally optimize an explicit cost function:

[

\mathcal{L}

\lambda1 L{\text{answer}}

• \lambda2 L{\text{latency}}

• \lambda3 L{\text{tool cost}}

• \lambda4 L{\text{incorrect reuse}}. ]

The weights express the system designer’s tolerance for errors, delay, and computational expense.

14. Latency improvements

The reported results demonstrate that streaming retrieval can reduce median first-token latency.

For Qwen2.5–7B, the reported p50 latency reduction is approximately:

• (0.58) seconds on synthetic audio;
• (1.80) seconds on human-spoken audio.

For OpusLM, the reduction is approximately:

• (0.42) seconds on synthetic audio;
• (1.40) seconds on human-spoken audio.

The larger gains on human speech are particularly consequential. A plausible explanation is that natural human utterances are longer, less compressed, and more disfluent than synthetic speech. These properties create a wider interval during which retrieval can execute concurrently with speech.

A reduction of one or two seconds may appear modest in offline benchmarking, but in interactive systems it is perceptually substantial. Conversational quality is highly sensitive to silence after a user finishes speaking. Removing even several hundred milliseconds can make the system feel more attentive and competent.

The median alone, however, is not sufficient for production assessment. Tail latency matters. A voice agent should also be evaluated at p90, p95, and p99 because occasional multi-second stalls can dominate users’ subjective experience.

15. Accuracy preservation

Latency reduction is only valuable if it does not degrade answer quality.

The reported accuracy results show the following pattern:

SettingQwen2.5–7BOpusLMNo RAG11.1%18.4%RAG after the final query33.8%21.2%Streaming RAG34.2%23.6%

The decisive comparison is between conventional final-query RAG and streaming RAG. Streaming retrieval achieves approximately comparable — and in these results slightly higher — accuracy while decreasing latency.

The minor accuracy improvement should not automatically be interpreted as proof that streaming is intrinsically more accurate. It may result from retrieval diversity, multiple candidate queries, dataset variance, or the reflector’s selection behavior. The stronger conclusion is that speculative retrieval does not appear to impose a material accuracy penalty in the evaluated setting.

The no-RAG baseline also illustrates the value of external evidence. For both models, retrieval substantially improves factual performance, although the magnitude differs by model.

Conclusion: the broader research problem

The most important contribution of Stream RAG is not one particular heuristic. It is the reformulation of retrieval for spoken systems as an incremental decision problem.

A conventional RAG system asks:

What should I retrieve after the complete question is available?

A streaming system asks:

At what earliest moment do I know enough to retrieve useful evidence, and when should I revise that retrieval?

This shift creates several research directions:

• estimating semantic completeness from partial speech;
• handling unstable transcripts and recognition revisions;
• determining whether new words alter the underlying information need;
• coordinating retrieval with interruption and turn-taking;
• minimizing redundant tool calls;
• predicting whether retrieved evidence will remain useful;
• learning latency-aware tool-use policies;
• exposing uncertainty when speculative retrieval may be wrong.

The larger production lesson is that major user-experience improvements often emerge from solving apparently small orchestration problems. The underlying language model may remain unchanged. The gain comes from deciding when to act, what to retrieve, which intermediate result to trust, and how to conceal computation behind the natural duration of human speech.

Stream RAG therefore represents more than an optimization of RAG latency. It is a step toward voice agents that reason and use tools continuously while a conversation is unfolding, rather than waiting passively for language to end before intelligence begins.