Building Voice-First Experiences with OpenAI Realtime API

The future of AI isn’t just about text — it’s about conversation. With the release of the OpenAI Realtime API, developers can now build fluid, low-latency voice agents that listen, understand, and respond like a human. This API powers the advanced voice mode in ChatGPT, and now it’s available for anyone to build with.

What Is the Realtime API?

Traditionally, building a voice assistant required stitching together multiple APIs:

1. Speech-to-Text (STT): Convert audio to text (e.g., Whisper)
2. LLM: Process the text with GPT-4 or similar
3. Text-to-Speech (TTS): Convert the response back to audio

This multi-step pipeline introduced latency and stripped away emotional nuance. The Realtime API solves this by using a single model (gpt-realtime) that handles audio input and output directly—no intermediate text required.

Technical Deep Dive

1. The WebSocket Connection

The Realtime API operates over WebSockets, allowing for a persistent, bi-directional stream of data. Unlike standard REST APIs, you don’t send one request and wait; you maintain an open “conversation” where audio chunks and text events flow back and forth simultaneously.

To start a session, you connect to the following endpoint: wss://api.openai.com/v1/realtime?model={realtime-model}

2. Session Configuration & Customization

Once connected, the first thing you typically do is send a session.update event. This is where you define the "soul" of your agent, including its voice, instructions, and tools.

• Voices: You can choose from built-in voices like alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, and cedar.
• Turn Detection: You can enable Server VAD (Voice Activity Detection), which allows the model to automatically detect when a user starts and stops speaking.

{
  "type": "session.update",
  "session": {
    "modalities": ["text", "audio"],
    "instructions": "You are a friendly travel guide. Speak with excitement and use short sentences.",
    "voice": "shimmer",
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "silence_duration_ms": 600
    }
  }
}

Streaming Audio: How it Works

When the user speaks, your client sends input_audio_buffer.append events containing base64-encoded audio chunks. The model processes these in real-time. When the model responds, it sends back response.audio.delta events—small packets of audio that you can play immediately on the client side, even before the model has finished its entire sentence.

This “streaming” of audio is what brings the latency down to sub-second levels, making it feel like a natural human conversation.

Use Cases

• Customer Support Bots: Replace frustrating “Press 1 for Sales” menus with natural conversations that can actually solve problems using Function Calling.
• Accessibility Tools: Enable voice-driven navigation for visually impaired users
• Multilingual Tutors: Real-time language learning with pronunciation feedback

Pricing Notes

• The Realtime API is a premium offering and is priced per token (both text and audio):
• Audio Input: ~$100 per 1 million tokens (roughly $0.06 per minute).
• Audio Output: ~$200 per 1 million tokens (roughly $0.24 per minute).
• Note: These prices are estimates based on average token density; check the official OpenAI pricing page for real-time updates.

Final Thoughts

The OpenAI Realtime API marks a shift from text-based chatbots to natural, Voice-first AI experiences. Whether you’re building assistants, tutors, or support agents, this API gives you the tools to create truly conversational interfaces.