[Part 2] Speech LLMs: How to Stream Audio?

Understanding WebSockets, JSON Payload and Base64 Encoder-Decoder.

In my previous guide, we covered the critical audio physics parameters — Sample Rate, Channels, and Chunk Size — needed to optimize speech for Large Language Models.

[embed][Part 1] Speech LLMs: Audio Configurations Explained: Sample Rate, Channels, and Chunk Size If you are building real-time voice agents or working with Speech Large Language Models (LLMs) like OpenAI’s Realtime…medium.com

But once your microphone is properly configured to capture clean 16 kHz mono audio, a major software engineering hurdle remains: How do you actually transport that audio data to an AI model in real time?

If you try to dump raw binary audio into modern native audio APIs like OpenAI’s Realtime API or Google’s Gemini Live, your connection will instantly break.

Building a real-time voice agent requires a specific web architecture. Here is how to bridge the gap between raw binary sound waves and text-based AI brains using WebSockets, JSON, and Base64.

1. Core Challenge: Binary Water in a Paper Envelope

Modern Speech LLMs do not just process audio; they engage in complex, multi-modal conversations. Over a single live connection, the system must handle user audio, text transcripts, system instructions, tool calls, and the AI’s spoken responses simultaneously.

To organize this multi-directional chaos, APIs rely on JSON to structure every message (or “event”) sent between the client and the server.

Here is the problem: JSON is strictly a text-only format. It can only read and write strings, numbers, booleans, and arrays. On the other hand, raw audio from a microphone is pure binary data — a dense stream of raw 1s and 0s.

Think of a JSON payload as a standard paper envelope. Think of raw binary audio as a puddle of water. You cannot pour raw water directly into a paper envelope without destroying it. You need a way to transform that water into a dry, stable package before mailing it.

2. Solution: Base64 Encoding

To safely transport binary audio inside a text-based JSON object, we use Base64 Encoding.

Base64 is a binary-to-text encoding scheme. It takes any stream of raw binary bytes and translates them into a safe string of 64 standard ASCII characters (letters A-Z, a-z, numbers 0–9, and symbols + and /).

When implementing a Speech LLM, Base64 acts as your universal translator at both ends of the conversation:

Input Pipeline (Encoding)

1. Your code captures a tiny network chunk of audio (e.g., 512 bytes of raw 16-bit PCM data).
2. Your local software feeds those 512 binary bytes into a Base64 encoder.
3. The encoder spits out a harmless text string that looks like this: UklGRiQAAABXQVZFZm10I...
4. You wrap that text string inside a JSON event object and safely send it across the network.

Output Pipeline (Decoding)

1. The AI model generates an audio response.
2. The server sends you a JSON object containing a long Base64 text string representing the AI’s voice.
3. Your code extracts that string and passes it through a Base64 decoder.
4. The decoder transforms the text back into raw binary PCM bytes, which your software instantly pumps into the user’s speakers.

💡 Is Base64 Lossless?

When you hear the phrase “converting audio,” it is natural to worry about quality degradation. Does converting my audio into text ruin the sound? No. Base64 is a 100% lossless process. Base64 is not a compression algorithm (like how MP3 compresses and permanently throws away audio data). It is simply a mathematical translation. You are taking base-2 math (binary 1s and 0s) and rewriting it in base-64 math using text characters.

Because it does not compress or remove any data, the Base64 text string is actually about 33% larger than the original binary file. However, because every single 1 and 0 is perfectly preserved in the translation, the audio you decode on the other side is an exact, bit-for-bit clone of the audio you originally captured.

3. Highway: WebSockets vs. HTTP

Now that your audio is safely wrapped in JSON text, how do you send it?

Traditional web applications rely on standard HTTP requests (like standard REST POST endpoints). If you send a message, you wait for the server to send a response back.

This model completely fails for conversational AI. If you have to wait for the user to finish their entire sentence, package it, hit a POST endpoint, and wait for the model to think, your voice agent will suffer from massive, unnatural delays.

Instead, Speech LLMs utilize WebSockets.

A WebSocket establishes a persistent, permanent, bi-directional highway between your application and the AI server.

• Connection is established once at the start of the call.
• Your application continuously streams tiny Base64 text chunks down the highway every few milliseconds.
• The AI server continuously receives them, analyzes them using server-side Voice Activity Detection (VAD), and can stream its own Base64 voice chunks back up the highway at any moment — even interrupting itself if it hears the user speak over it.

4. Step-by-Step Audio Streaming Pipeline

When you pull all of these concepts together alongside our audio configurations from Part 1, the actual real-time streaming pipeline looks like this:

Phase 1: User Speaks (Input)

[Microphone] 
     │  Captures raw sound wave
     ▼
[16 kHz, Mono PCM Audio Buffer] (e.g., 512 Bytes / 16ms of sound)
     │  
     ▼
[Base64 Encoder] 
     │  Converts binary bytes to an alphanumeric text string
     ▼
[JSON Object Wrapper] -> { "type": "input_audio_buffer.append", "audio": "UklG..." }
     │  
     ▼
[WebSocket Client] ───( Streamed instantly over the internet )───► [AI Server]

Phase 2: AI Responds (Output)

[AI Server] ───( Streamed instantly over the internet )───► [WebSocket Client]
                                                                  │
                                                                  ▼
[JSON Object Wrapper] <- { "type": "response.audio.delta", "delta": "SGVs..." }
                                                                  │  Extracts string
                                                                  ▼
                                                           [Base64 Decoder]
                                                                  │  Converts text to binary
                                                                  ▼
                                                    [24 kHz, Mono PCM Audio Buffer]
                                                                  │  
                                                                  ▼
                                                             [Speakers]

By separating your network transport layers (JSON over WebSockets) from your encoding layers (Base64), you gain total control over real-time streaming. Your code handles data logistics quickly and cleanly, leaving the native audio AI model free to do what it does best: understand and speak human language perfectly.