Adding Voice to a Java AI Assistant — Whisper, TTS, and the Voice Conversation Loop

How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform

Where We Left Off

After Phase 4, Jarvis could answer questions using real tools.

You: What is the weather in Kathmandu?
Jarvis: [calls WeatherTool] It is 22°C and sunny.

You: What is 2847 × 391?
Jarvis: [calls CalculatorTool] 1,113,177

But every interaction required typing.

Phase 5 changed that.

The Goal

BEFORE Phase 5

You type
      ↓
Jarvis types back

AFTER Phase 5

You speak
      ↓
Whisper transcribes
      ↓
AI responds
      ↓
TTS speaks back

Simple to describe.

Surprisingly nuanced to build correctly.

The First Surprise — Ollama Does Not Support Whisper

The original plan was to run Whisper locally through Ollama.

ollama pull whisper

Error:
pull model manifest: file does not exist

Ollama is excellent for language models.

It does not support speech transcription models.

That forced a complete redesign.

The Solution — Two Modes

We designed WhisperTranscriptionService to support two interchangeable backends.

Mode 1 — Groq API

Groq provides Whisper large-v3-turbo through an OpenAI-compatible API.

The free tier offers approximately 6,000 requests per day, making it ideal for development.

Setup is as simple as adding:

GROQ_API_KEY=your-key

to your .env file.

Mode 2 — Local whisper.cpp

For users wanting a fully local solution:

git clone https://github.com/ggerganov/whisper.cpp

cd whisper.cpp

make

bash ./models/download-ggml-model.sh base.en

./server -m models/ggml-base.en.bin --port 8178

Both implementations expose the same OpenAI-compatible multipart API.

Switching between them is just a configuration change.

The Most Important Design Decision

The biggest architectural decision of Phase 5 was surprisingly simple.

Voice is not a separate AI pipeline.

Voice is simply another input and output layer around the existing system.

Instead of this:

Audio
  ↓
Different AI Pipeline
  ↓
Different Memory
  ↓
Different Tools

we built this:

Audio
  ↓
Whisper
  ↓
Text
  ↓
AiOrchestrator.chat()
  ↓
Existing Memory
Existing RAG
Existing Tools
  ↓
Text
  ↓
Text-to-Speech

Nothing inside the AI pipeline changes.

Everything built during Phases 1–4 automatically works for voice conversations.

WhisperTranscriptionService

The service performs three responsibilities:

• Validate audio input

• Determine whether Groq or whisper.cpp should be used

• Execute blocking transcription safely on Schedulers.boundedElastic()

Running transcription on the WebFlux event loop would block every request.

Moving it to boundedElastic() keeps the application responsive while the HTTP request completes.

A second design choice was isLocalMode.

When local transcription is enabled, no API key is required.

The same service class handles both deployment models.

Text-to-Speech Without Extra Dependencies

For speech synthesis we deliberately avoided third-party Java libraries.

Instead we rely on each operating system’s built-in speech engine.

Windows

• PowerShell + System.Speech.Synthesis

macOS

• say

Linux

• espeak / text2wave

Advantages:

• Zero additional dependencies

• No API keys

• Works immediately

• Offline

• Cross-platform

Voice selection and playback speed are controlled entirely through environment variables.

DST Awareness — A Tiny Bug That Wasn’t Tiny

During review we discovered a subtle timezone issue.

Using

getDisplayName(false, ...)

always reports Standard Time.

During summer this produces incorrect timezone names for regions observing daylight saving time.

The fix was to determine whether the current instant is actually inside a DST period before generating the display name.

One small boolean made timezone names correct year-round.

The Sentence Buffering Problem

Language models stream tokens.

"The"

"weather"

"in"

"London"

"is"

"22"

"°"

"C"

"and"

"sunny"

"."

Reading those tokens individually sounds awful.

Instead we buffer tokens until either:

• a sentence finishes

or

• fifty words accumulate

Only then is the sentence sent to the speech engine.

Three implementation details matter.

First, concatMap() guarantees sentences are spoken sequentially.

Using flatMap() would cause overlapping speech.

Second, the 50-word safety limit prevents infinitely growing buffers when models omit punctuation.

Third, speech generation runs on boundedElastic() so audio generation never blocks streaming responses.

Two Independent Pipelines

The first implementation looked like this.

Token
  ↓
TTS
  ↓
Next Token

Users wouldn’t receive the next token until speech playback finished.

The experience felt slow.

The final architecture separates streaming and speech.

Token Stream
       │
┌──────┴────────┐
│               │
▼               ▼

Browser SSE Sentence Buffer

Immediate Background

│               │
▼               ▼

Live UI Text-to-Speech

The browser begins rendering almost immediately.

Speech begins as soon as the first complete sentence is available.

Neither pipeline blocks the other.

VoiceChatEvent

The SSE endpoint emits structured events instead of plain strings.

The first event contains the newly created session ID.

Subsequent events stream generated tokens.

This allows clients to continue future voice conversations using the same session without any additional API calls.

Voice REST API

The voice system exposes five endpoints.

POST /api/v1/voice/transcribe

Transcribes uploaded audio.

POST /api/v1/voice/speak

Plays synthesized speech directly on the server.

POST /api/v1/voice/speak/bytes

Returns WAV audio for browsers and desktop clients.

POST /api/v1/voice/chat

Streams an entire AI conversation from spoken input.

GET /api/v1/voice/status

Reports whether transcription and speech are currently available.

Lessons Learned

Ollama does not support audio models.

That assumption was incorrect.

Community feedback caught it before implementation.

Every blocking operation must be isolated.

Whisper requests.

System speech commands.

Audio generation.

Everything belongs on Schedulers.boundedElastic().

festival --tts cannot generate WAV files.

Linux audio generation requires text2wave or Festival's Scheme API.

Timeout handling matters.

When speech generation exceeds the configured timeout, child processes must be destroyed or they continue running indefinitely.

Timezone handling is surprisingly difficult.

Correct timezone names require evaluating daylight saving time for the current instant rather than assuming standard time.

Voice Status Endpoint

Before enabling voice, clients can verify the environment.

Example response:

{
  "success": true,
  "data": {
    "transcriptionAvailable": true,
    "ttsAvailable": true,
    "voiceReady": true,
    "transcriptionMode": "groq-cloud",
    "ttsEngine": "system-macos"
  }
}

This immediately tells users whether they’re using Groq or local Whisper and which speech engine has been detected.

A Complete Voice Conversation

User speaks

"What is the weather in Kathmandu?"

        │
        ▼

Whisper
(Groq / whisper.cpp)

        │
        ▼

AiOrchestrator.chat()

    ├── Session History
    ├── Long-Term Memory
    ├── RAG Context
    └── Tool Calling

        │
        ▼

WeatherTool

        │
        ▼

AI Response

    ├────────► Browser (SSE)

    └────────► Text-to-Speech

The important observation is that nothing inside the AI pipeline changes.

Voice simply wraps the architecture built during Phases 1–4.

What’s Next

Phase 6 introduced the Agent System, allowing Jarvis to plan and execute multi-step tasks autonomously.

Phase 7 brings a complete web interface built on top of everything developed so far.

The backend is now complete.

Phases 1–6 are merged, tested, and production-ready.

Jarvis can now hear.

Jarvis can now speak.

Contributing

Jarvis is open source under the Apache 2.0 License.

Current contributor-friendly issues include:

• #69 — CLI Voice Commands

• Voice Integration Tests

GitHub

github.com/sujankim/jarvis-ai-platform

Jarvis AI Platform Series

Part 1 — Building a Local-First AI Assistant with Spring Boot 4

Part 2 — Building Long-Term Memory with pgvector

Part 3 — Implementing Semantic Memory Retrieval

Part 4 — Building a Tool Engine with Spring AI

Part 5 — Adding Voice with Whisper and Text-to-Speech (this article)

Part 6 — Building an AI Agent System with the ReAct Pattern (coming next)

Your AI. Your Data. Your Machine.