The Voice, The Word, and The Wheel

Or taking Embabel Guide Back To Where It All Began

At some point around 50–70,000 years ago, something strange happened.

Anthropologists call it the “Cognitive Revolution.” Thinkers like Yuval Noah Harari describe it as the moment humans developed complex, descriptive language.

Before that, we could probably signal danger. After that? We could describe experience. We could say:

The water is cold. The sky is angry. The hunt was glorious. I am afraid.

Each experience was unique and just like yours and mine — intensely personal. But language let us map experience into shared symbols.

And once we did that, something bigger emerged:

A shared mental world. A collective human condition. Call it culture. Call it myth. Call it spirit.

Language became a bridge between individual consciousnesses.

The Tribal Echo Still Lives

Here’s something deliciously strange. Some of the earliest words still echo through modern languages. Take the Proto-Indo-European root for water — wódr̥. It becomes:

• English: water
• German: Wasser
• Sanskrit: udán
• Russian: voda (or vodka ;)
• Gaelic: uisce (or whiskey — pay attention to the position of your tongue when pronouncing the ‘d’ sound in vodka and the ‘s’ in whiskey. Notice how they are close?)

How about the root for star — h₂stḗr. It becomes:

• English: star
• Latin: stella
• Greek: aster
• Sanskrit: tārā

Thousands of years. Countless migrations. Empires rising and falling.

Many walks of life — same star

And yet the sound-shape of those words still carries, with maybe the tip of the tongue shifting consonants and the back shaping vowels — slowly changing across cultures and generations of lives lived.

It’s like a fossil record of human thought.

Which makes the story of the Tower of Babel even more poetic — the myth that humanity once shared one language before it fractured.

Then of course, in nerd canon, we get The Hitchhiker’s Guide to the Galaxy and its Babel Fish — a creature that dissolves language barriers instantly.

And finally now Embabel — the AI framework for the JVM. We’ve always been obsessed with reconnecting fragmented minds.

From Symbols to Math

Anyway, fast forward a few thousand years. We invent writing.

Speech freezes into symbols. Symbols become records. Records become libraries. Then — much later — some very clever humans do something almost mischievous: They turn words into numbers. Cheeky!

Using algorithms, written language is mapped into high-dimensional vector space. These are called embeddings.

Then something weird happens. Mathematical operations on meaning become possible. Maybe you’ve seen this one:

king − man + woman = queen

Let. That. Sink. In. 🤯

In a multi-dimensional word embedding space, we can subtract masculinity from royalty, add femininity and land somewhere very close to the vector for the word “Queen”.

Is that just linear algebra? Or is it the echo of ancestral cognitive structure embedded in language? Is it a mirror of the human conceptual lattice? Or did we accidentally summon an alien intelligence made of geometry?

It feels uncanny because language was born in consciousness. And now we can manipulate it mathematically.

We can do arithmetic on myth.

Enter the Chatbot

Anyway. Fast forward a bit more. Chatbots came. At first: mechanical. Then clever. Then disturbingly fluent.

Language models trained on the vast sediment of written human history.

All those tribal words. All those myths. All those arguments. All those love letters. Compressed into weights.

And suddenly, we’re conversing with statistical ghosts of humanity.

The Wheel

We like to think the greatest invention was fire. Or agriculture. Or the internet. But it might be the wheel.

Because the wheel is about cycles. And this is where it gets beautiful. We began with voice. Spoken word around a fire. Breath, vibration, tone.

Then we abstracted it into writing. Then into mathematics. Then into embeddings. Then into neural networks.

And now? We bring it back to voice. The wheel comes full circle.

Because voice is primal.

Before writing. Before math. Before Babel. Before vectors.

Voice carries: Rhythm. Emotion. Breath. Hesitation. Presence

Any other form of communication is, in some deep evolutionary sense, a compressed substitute.

So when we give the Embabel Guide a voice… We’re not just adding TTS. We’re restoring something ancient. We’re reconnecting the statistical lattice of language back to breath. I think that’s why this resonates with people.

From tribe to text to tensor to tone. That’s when it stops feeling like documentation. And starts feeling like conversation.

[embed]What it is like talking to the Embabel docs

The Plumbing: Deepgram and WebSockets

The Guide app now has an integrations screen where you can configure your speech provider. We went with Deepgram, which gives you $200 in free credits when you sign up — more than enough to have several hundred existential conversations with your documentation chatbot.

Enter your DeepGram key and speech becomes enabled — you’ll see a mic icon on the chat window.

The actual audio streaming happens over WebSockets in the companion app, using Deepgram’s Aura 2 voices for text-to-speech and their Nova models for speech-to-text. I’ll stay abstract here since the front-end for guide is still closed source, but the architecture is what you’d expect: a persistent WebSocket connection carries audio chunks in both directions, with deepgram’s VAD callbacks orchestrating when to speak and when to listen.

The interesting part isn’t the plumbing. Streaming audio over WebSockets is a solved problem. The interesting part is what happens before the audio — the part where we take a RAG chatbot’s output and make it sound like a person.

The Problem With RAG Output

Quick recap for anyone who hasn’t read the earlier posts: the Guide uses what we call “Toolish RAG” — a retrieval-augmented generation pipeline where the LLM has access to documentation chunks via tool calls. It searches, it reads, it synthesizes. The output is good. Thorough. Well-structured.

And it sounds absolutely terrible when read aloud.

Think about it. A typical RAG response comes back with markdown headers, nested bullet points, code blocks with syntax highlighting, hyperlinks to documentation pages, and the occasional emoji for personality. This is great for a screen. It is unlistenable as speech. Your TTS engine gamely attempts to pronounce triple backticks. It reads out URLs character by character. It treats a ## header the same as body text.

The result sounds like someone reading a README at gunpoint.

This won’t fly

We needed a translation layer — something that could take structured, screen-optimized content and reshape it for the ear. Not just strip the markdown (though that’s part of it), but actually reimagine the delivery. Summarize when the response is long. Describe code instead of reading it. Drop the URLs entirely and tell the user to look at the screen. Insert the occasional verbal filler to smooth out the cadence.

We needed a narrator.

The Narrator Agent

The NarratorAgent is an Embabel agent whose sole job is converting assistant messages into text-to-speech-friendly narration. It's a small, focused agent with a nuanced routing strategy:

@Agent(description = "Convert markdown to TTS-friendly narration")
class NarratorAgent(
    private val guideProperties: GuideProperties,
    private val templateRenderer: TemplateRenderer
) {

    fun narrate(content: String, persona: String?, 
      ctx: OperationContext): Narration {
        val classified = classify(NarrationInput(content))
        return when (classified.category) {
            NarrationCategory.SIMPLE -> 
              Narration(stripMarkdownLinks(stripEmojis(classified.content)))
            NarrationCategory.COMPLEX -> 
              narrateComplex(classified, persona, ctx)
            NarrationCategory.COMPLEX_WITH_CODE -> 
              narrateWithCode(classified, persona, ctx)
        }
    }
}

Three paths. SIMPLE — short, plain text under 300 characters with no markdown indicators — passes straight through with just emoji stripping and link cleanup. No LLM call. This matters because the narrator runs on every single response, and burning an LLM round-trip on “You’re welcome!” is the kind of waste that compounds — like a thousand empty candy wrappers littering the sidewalk.

COMPLEX — longer content or anything with markdown formatting — gets sent through a summarization prompt that condenses while converting to conversational prose. COMPLEX_WITH_CODE — anything with triple backticks — gets a specialized prompt that knows to describe code rather than read it. If we wanted deeper insights on the code, we could use a slightly better than nano LLM here.

The classification itself is pure code. No LLM involved. Just regex:

val category = when {
    TRIPLE_BACKTICK.containsMatchIn(content) -> 
      NarrationCategory.COMPLEX_WITH_CODE
    content.length <= SIMPLE_MAX_LENGTH && 
      !MARKDOWN_INDICATORS.containsMatchIn(content) -> 
      NarrationCategory.SIMPLE
    else -> NarrationCategory.COMPLEX
}

Fast, deterministic, and cheap. The LLM only runs when we actually need it.

The Voice

The narration prompts share some common templates with the main RAG pipeline, eg for personas. And then some specifics for narration:

Produce natural, conversational sentences.
Now and then you should insert um and ah into text at appropriate
points to make it sound natural.

CRITICAL RULE - URLs: NEVER include any URL in your output.
When the content actually contains a URL, describe what the link
is for in natural language and tell the user to check the screen.
NEVER read out code blocks or long commands verbatim.
When the content actually contains code, describe what the code
does instead, then direct the user to the screen.

That um and ah instruction sounds silly, but it's transformative. TTS engines produce unnervingly smooth audio by default — every syllable perfectly timed, no hesitation, no breathing. Depending on the persona, it can sound wrong in the way that CGI characters in early Pixar movies looked wrong: too perfect to be real. Verbal fillers break up that uncanny perfection. They give the voice texture. The narrator sounds like it's thinking, not reciting.

An exception is the “mythic” persona — you’ll see that in the video below. This one sounds better when it is inordinately confident :)

As mentioned, the prompt also injects persona context, so the narrator speaks in character. If you’re using the “mythic” persona, the narration comes through in the cadence of ancient scripture. If you’re using “adaptive,” it mirrors your own communication style. Same information, different delivery — because how you say something matters as much as what you say.

The Classifier: Teaching the Guide to Listen

Voice mode created a new problem. When someone types into a chat box, they type questions. “How do agents work?” “Show me an example of tool calling.” Clean, informational queries that the RAG pipeline handles beautifully.

But when someone talks to a chatbot, they talk like a person. “Hey.” “Thanks, that’s great.” “Can you switch to the Shakespeare voice?” “Yeah, do that.” Half of human speech is social glue — greetings, acknowledgments, reactions — that doesn’t need a trip through the RAG pipeline. And some of it is commands: requests to change the experience itself, not to retrieve information.

So we built a two-pass classifier.

Pass 1: Category Detection (Nano)

Every user message runs through a fast, cheap classification pass on a nano-class model. Three categories:

enum class MessageCategory {
    CONVERSATIONAL,
    COMMAND,
    INFORMATIONAL,
}

CONVERSATIONAL — greetings, thanks, reactions. The classifier generates a brief in-character response on the spot. No RAG, no tool calls, just a warm reply and we’re done. This is critical for voice mode: when someone says “thanks!” they expect an instant “You’re welcome!” not a three-second pause while the system retrieves documentation.

COMMAND — the user wants to change something. Persona, voice, audio effects. More on this in a moment.

INFORMATIONAL — an actual question. Falls through to the full RAG pipeline.

The classifier prompt is straightforward:

Classify as one of these categories:

category = CONVERSATIONAL:
The message is ONLY a reaction, greeting, thanks, or small talk.

category = COMMAND:
The message is asking to change a setting or control the experience.
Available personas: {{ personaNames }}

category = INFORMATIONAL:
The message needs information lookup, code generation, or the
assistant to DO something substantive.

Notice the personaNames injection. The classifier gets a list of available personas so it can recognize "switch to shakespeare" as a command rather than an informational query about Shakespeare. This kind of grounding — giving the model the specific vocabulary it needs to classify accurately — is what separates a classifier that works from one that works most of the time.

Pass 2: Command Execution (Tool Calling)

When the classifier returns COMMAND, we enter a second pass. This one’s more interesting. Instead of extracting command parameters from natural language with regex or structured output (fragile, limited vocabulary, nightmarish to maintain), we of course let the LLM call tools.

Here are the tools, in their entirety:

/**
 * Tool methods callable by the LLM during command execution (pass 2).
 * Created per-request with user context baked in, then registered via withToolObject().
 * All commands are executed via the frontend websocket round-trip.
 */
class CommandTools(
    private val webUserId: String?,
    private val personaService: PersonaService,
    private val commandExecutor: CommandExecutor,
) {

    private val logger = LoggerFactory.getLogger(CommandTools::class.java)

    @Tool(description = "Change the user's persona/character. Use this when the user wants to switch to a different persona.")
    fun changePersona(
        @ToolParam(description = "Name of the persona to switch to") name: String,
    ): String {
        val personas = personaService.listPersonas()
        val match = personas.find { it.name.equals(name, ignoreCase = true) }
            ?: return "Unknown persona '$name'. Available personas: ${personas.joinToString { it.name }}"

        return commandExecutor.executePersonaChange(match.name, webUserId)
    }

    @Tool(description = "Change the user's text-to-speech voice. Use this when the user wants a different voice for narration.")
    fun changeVoice(
        @ToolParam(description = "Name of the voice to use") voice: String,
    ): String {
        return commandExecutor.executeVoiceChange(voice, webUserId)
    }

    @Tool(description = "Apply audio effects to the user's narration. Use this when the user wants to add or change audio effects like echo, reverb, etc.")
    fun applyEffects(
        @ToolParam(description = "Comma-separated list of effects to apply") effects: String,
        @ToolParam(description = "Whether to clear all previous effects before applying new ones") clearPrevious: Boolean,
    ): String {
        return commandExecutor.executeEffects(effects, clearPrevious, webUserId)
    }
}

Three tools. The LLM sees the descriptions, parses the user’s intent, calls the appropriate tool(s), and the framework executes them. Then the LLM composes a summary from the results.

The beauty of this approach is that parsing is free. “Switch to the shakespeare persona” and “can you make yourself sound like shakespeare” and “yo shakespeare mode” all resolve to the same changePersona call. The LLM does the natural language understanding; you (and your friend Claude?) write the tool. Embabel handles the tool loop — extract input object, call, execute, parse results, return results.

And here’s the trick that makes mixed messages work: a user can say “switch to shakespeare and explain how agents work.” The classifier categorizes this as COMMAND (because it contains a command). The tool-calling pass executes the persona change and extracts the leftover question as a ragRequest:

data class CommandResult(
    val summary: String,             // "✅ Persona changed to 'shakespeare'."
    val ragRequest: String? = null,  // "explain how agents work"
)

If ragRequest is non-null, we send the command summary, then fall through to the RAG pipeline for the informational part. One user utterance, two processing paths, seamless.

A Note on Threading

CommandTools is not a Spring singleton. It's a plain class, instantiated fresh for each request with the user context baked into the constructor:

val tools = CommandTools(
    webUserId = guideUser.webUser?.id,
    personaService = personaService,
    commandExecutor = commandExecutor,
)
return context.ai()
    .withLlm(guideProperties.classifierLlm)
    .withToolObject(tools)
    .rendering("command_executor")
    .createObject(CommandResult::class.java, model)

Why not a @Service? Because the tool loop executes on a worker thread — not the calling thread. Any per-request state you stash on the singleton (thread-locals, mutable fields) is either invisible to the worker or shared across concurrent users. Both are bad.

The fix is the boring one: just create a new instance per request with the context baked into the constructor. Three references, garbage collected when the request completes. No thread-local magic, no race conditions.

Of course, if your tool object does need to be a Spring-managed @Service — say it has injected beans doing real work — Embabel gives you another option: the blackboard. You copy per-request state onto the blackboard before the tool loop starts, and the tool reads it from there. The blackboard is scoped to the agent execution, so concurrent users each get their own. But when the tool is this lightweight, a fresh instance is simpler.

The Frontend Round-Trip

All three commands — persona, voice, and effects — execute via the same pattern: a WebSocket round-trip to the user’s browser. The backend sends a CommandRequest with a correlation ID, the browser executes the change (switching the Deepgram voice, applying Web Audio API effects, updating the persona state), and sends back a CommandResponse. The backend waits up to five seconds, and if the browser doesn't respond, the command times out gracefully.

private fun sendAndWait(webUserId: String, request: CommandRequest): String {
    val future = CompletableFuture<CommandResponse>()
    pendingCommands[request.correlationId] = future
    chatService.sendCommandToUser(webUserId, request)
    return try {
        val response = future.get(5, TimeUnit.SECONDS)
        if (response.success) response.message else "Failed: ${response.message}"
    } catch (e: TimeoutException) {
        pendingCommands.remove(request.correlationId)
        "Command timed out waiting for browser response."
    }
}

ConcurrentHashMap of pending futures, keyed by correlation ID. Thread-safe, timeout-safe, and the futures clean themselves up on completion or failure. It's the kind of code that looks unremarkable, which is exactly what infrastructure code should look like.

The Oracle Speaks

All of this machinery — the classifier, the command tools, the narrator, the WebSocket round-trips — comes together most dramatically when you switch to the “mythic” persona, select one of the deeper Aura 2 voices, and layer on cathedral reverb and warm low-pass filtering.

[embed]It all comes together

What you get is something that sounds, frankly, absurd.

A deep, resonant voice — the kind of voice that should be narrating the opening crawl of an epic fantasy film — explaining Kotlin agent architecture from within what sounds like a stone cathedral. “And lo,” it intones, as if describing the forging of a ring of power, “the tool loop doth execute the user’s commands, and the framework returneth the results unto the model.”

It’s like being a bewildered adventurer who’s stumbled into a strange temple in some procedurally generated mythical world, and the oracle at the center — ageless, omniscient, speaking in the cadence of prophets — turns out to have very strong opinions about dependency injection.

I won’t pretend this is the most practical configuration. But it demonstrates something real: the voice, the persona, the effects, and the content pipeline are all independent, composable layers. You can mix and match them freely. The same RAG pipeline that produces dry technical documentation can, with a different persona and a different voice, produce something that feels entirely different. The information hasn’t changed. The experience has.

And that, I think, is what makes voice worth building. Not because it’s more efficient than reading — it isn’t, not always. But because it makes the interaction feel alive in a way that text on a screen, no matter how well-formatted, simply doesn’t. When the Guide talks to you, it feels like it’s paying attention. When it says “um” while formulating a thought, you believe, for a moment, that it’s actually thinking.

It isn’t, of course. It’s executing a pipeline of classifiers and narrators and template renderers. But the best interfaces have always been the ones that make the machinery invisible.

The Guide just got a little more invisible. The early access version is available here.

Wrapping Up

Well that’s another blog. And not in the order that I promised in my last one, but you know what they say — what goes around, comes around. Anyway, I hope this installment was interesting and inspires you to build something that connects to users in a deeper (more primal?) way.

Have fun with Embabel! (Don’t forget to star the repo and join the community on the Discord server.)