Why I Built a Custom Voice Activity Detector in Java

In late August 2024, I started building a conversational AI voice application — the kind where you call a phone number, an AI answers, and you have a back-and-forth conversation. The stack was Java, Spring Boot 3, and Twilio for the telephony layer.

The architecture seemed straightforward. Twilio streams audio from the caller over a WebSocket, my server processes it, sends it to a speech-to-text service for transcription, feeds the transcript to an LLM, converts the response to speech, and streams it back. A loop.

Except for one piece I had not figured out: how does the system know when the caller is done speaking?

The problem with continuous audio streams

Twilio does not send pre-segmented utterances. It sends a continuous stream of 8kHz mu-law encoded audio chunks over a WebSocket — about 20 milliseconds at a time, indefinitely. Silence, speech, background noise, the AI’s own response playing through the caller’s speaker — it all comes through as one undifferentiated river of bytes.

Without knowing where speech starts and ends, you cannot have a conversation. Send the audio to transcription too early and you cut the caller off mid-sentence. Wait too long and the interaction feels sluggish. Get it wrong consistently and the whole thing falls apart — the AI talks over people, responds to silence, or sits there waiting while the caller wonders if the line went dead.

I needed a voice activity detector. Something that could sit in the audio pipeline, listen to the stream in real time, and signal when speech starts and when it ends.

The cost of streaming everything

My first instinct was to let the transcription service handle it. Google Speech-to-Text, Deepgram, and OpenAI Whisper all offer streaming transcription with their own endpoint detection. They listen to the audio and decide when someone stops talking.

The issue is economics.

In a typical AI voice conversation, the caller does not speak for most of the call. The user asks a question — a few seconds. The system processes it — LLM latency. The AI responds — text-to-speech playback, which can run 10 to 30 seconds for a detailed answer. During that entire response, the audio stream is still flowing. It is just silence, or the AI’s own voice bleeding through the caller’s microphone.

In a 2.5-minute conversation I measured, the caller spoke for about 45 seconds total. The rest was system latency and AI responses. That is 30% speech, 70% billable silence.

If you stream all of that to a transcription API, you pay for 2.5 minutes. If you detect speech boundaries yourself and only send the actual speech, you pay for 45 seconds. A 70% cost reduction per call.

Without a VAD, you pay for the full call duration. With one, you pay only for actual speech.

The math gets more dramatic as conversations get longer. In many use cases — appointment scheduling, order status checks, simple confirmations — the user might say nothing more than “yes,” “Tuesday at 3,” or “that’s correct.” Five seconds of speech in a two-minute call. You would be paying for 115 seconds of silence.

At scale, this is the difference between a viable product and one that bleeds money. A thousand 5-minute calls per month where the user speaks for 1.5 minutes each: without a VAD, you are billed for roughly 83 hours of audio. With a VAD, 25 hours. The per-minute rate — whether it is Google’s $0.016, Deepgram’s $0.0077, or Whisper’s $0.006 — matters less than the minutes you do not send.

Cost comparison across transcription providers at 1,000 calls per month.

The Java VAD landscape — and what came next

With the economics clear, I went looking for existing Java solutions. What I found was thin — JNI wrappers around native code, heavyweight speech recognition frameworks, audio libraries oriented toward music analysis. None of them fit a well-scoped problem: take a stream of 8kHz telephone audio, figure out when someone is talking, and signal when they stop.

The full post covers building the VAD from scratch over three months of iterative development with AI assistance — from a simple energy threshold detector that worked 90% of the time, through the false positives that broke it (keyboard clicks, elevator doors, passing cars), to the multi-feature system that solved them: zero-crossing rate, spectral centroid, pitch detection through autocorrelation, and adaptive calibration that adjusts to ambient noise on every call.

👉 Read the full post at lebedev.ai/writing/java-voice-activity-detector/

This is Part 1 of a series on building real-time voice activity detection in Java. More at lebedev.ai/writing/