The Weeknd’s Starboy lyrics as misheard by a 4-bit quantized Whisper model. (I’m a motherfg Starboy)*

Quantize Karaoke is the Whisper AI Game You’re Missing

One of the projects I’ve been presented during this Asian sabbatical was custom use cases for Whisper.cpp. This is the open source C++ version of OpenAI’s Whisper audio transcription using Georgie Gerganov’s GGML. I get two questions about Whisper in the IoT space:

1. How good is it?
2. How small can we make this for low power edge devices?

Whisper is incredible at live transcribing audio in C++ even running on low power CPUs. I’ve demonstrated the spooky accuracy it has with the standard fp16 full-quality models. The compute and memory requirements usually aren’t a problem on a basic laptop but what about a small RaspberryPi Cortex A series or even MCU with limited RAM? I set out to demonstrate the minimum viable speech recognition environment for potentially solar powered kiosks in the field. Ever since the YouTube sensation recognizing the word “Kubernetes” as “Cuban 80s” the world is nostalgic for audio transcription that isn’t perfect. Let’s introduce some imperfection into transcribed audio using quantization and temperature.

The value to Whisper is the models provided by OpenAI. These are trained on 680,000+ hours of audio/transcription data and are incredibly effective for not just speech-to-text but general transcription. I’ve shown in spooky demos when an ambulance drives by outside while you’re live transcribing it adds an [an ambulance drives by] into the live transcript. The Whisper.CPP project creates a performant fork of this inference runtime in pure C++ with GPU support optional. You can even try the live examples on a phone or mobile device with WebAssembly.

The key to making this work in low-power, low-RAM environments is model quantization. There are a few sizes of models available in GGML formatted native FP16. These models are trained on different amounts and quality of source data. Base, tiny, small, medium, and large are available in different versions and some English-specific. Tiny is a respectable 77MB. That transcribes pretty well on laptop hardware but to make it work well on a low-powered SBC at scale it’s possible to cheat a bit by quantizing the models to lower formats from 16-bit floating point. This means losing detail down to 8-bit, 4-bit, or even 3-bit. As you might guess this can quarter the memory footprint of a model. Depending on how the compute scales vectors it can also simplify the vector math for transcription. This vector scaling and repacking is something Nvidia’s new Blackwell series GPU is superb at, able to handle 64,32,16,8, and 4 bit scalars and vectors in a way that maximizes all resources in parallel and doesn’t require resizing.

What happens when using a quantized model at low detail? The best benchmark I’ve found for this is simply trying to transcode music. It’s a perfect test as it contains background noise, bizarre pronounciation, and audio that would challenge anybody with Auditory Processing Disorder which quantized models can simulate. In fact in the 90s every vowel sounded like “A” for some reason. Musical lyrics are often so poetic that context isn’t consistent which also throws off the transcription which occasionally can backtrack and adjust words it already transcribed based on words that come later affecting the meaning of prior words.

Oh Gotye.. “I guess that I don’t need that though.” Somebody That I Used to Know

The result with using lower detailed models with music is some kind of Weird Al misinterpretation of audio lyrics. It can actually be fun sometimes, and I tried a Malaysian karaoke night where we let a 4-bit tiny quantized model provide lyrics for a live karaoke night. The ffmpeg process go something like this. I started with The Weeknd’s song Starboy:

1. Take the music video file.
2. Pass the audio through whisper-cli, outputting a subtitle file.
3. Re-pack the video file with whisper’s subtitles.

Taking things a step further with GGML temperature randomization outputs all kinds of variants:

ffmpeg -hide_banner -i video.mp4 -acodec pcm_u8 -ar 41000 sound.wav
whisper-cli -tp 0.95 \
 -f sound.wav \
 -m ~/code/whisper.cpp/models/ggml-tiny.q4_0.bin \
 -osrt true
ffmpeg -hide_banner -i video.mp4 -i video.wav.srt \
 -map 0:v -map 0:a -map 1:s \
 -c copy output.mkv

“I’ll come alive in the fall-time.”

I tried to break things a bit more with the song One Week by Barenaked Ladies. It turns out this actually works pretty well and the lyrics are articulate enough for even quantized models to pick up. Just a few defects come across:

“Can’t understand what I mean well you soon will.”

I’ll set up a folder with some output subtitle files but in general I think Whisper is still rock solid using full models and also works perfectly acceptable in low-powered IoT microphones when there is little background noise. I don’t think courtroom stenography is dead quite yet. Even full quality models sometimes have trouble with song lyrics. When it comes to live television subtitle transcription though, Whisper was a real game changer and it put quite a few people out of work early on.

Sometimes just “Music” instead of Doja Cat’s “Let me be your woman.”

Other times “let me be your Roomba.” By Gemini Pro v2.5

How good is Whisper.cpp? It’s as good as the resources you let it use, and often this is excellent but not quite perfect. In language and accents there is a lot of room for error. Is minimal Whisper good enough for kiosks in a quiet space? Absolutely. If there will be music or background noise then the use case probably requires full-detail models.

[embed]GitHub - ggml-org/whisper.cpp: Port of OpenAI's Whisper model in C/C++ Port of OpenAI's Whisper model in C/C++. Contribute to ggml-org/whisper.cpp development by creating an account on…github.com

One thing I’ve done is wrap the library in an extended C++ std::streambuf which allows the seamless replacement of any std::istream in your app with a live audio device as input:

std::string sample;
std::istream *input = &std::cin;
WhisperBuf microphone;
std::istream whisperStream(&microphone)

if (!microphone.start())
  exit (1);

// Seamlessly switch input from stdin to Whisper audio transcription.
input = &whisperStream;
std::cout << "What is your input? ";
*input >> sample;

Instantly any C++ app that reads from a file or standard input can now read from your voice instead. This example uses default options including the default SDL2 audio input device and default model. Funny enough I’ve also used this as a coding challenge for many of the latest LLM coding models. Most couldn’t replicate this correctly for me until ChatGPT 4o came along.

[embed]ggml AI inference at the edge. ggml has 12 repositories available. Follow their code on GitHub.github.com

If anyone has questions about how to make use of Whisper.cpp please do ask or else go directly to the new GGML org for support. It really is a powerful inference engine whether been applied for accuracy or applied for playful use cases like Quantize Karaoke.