Revisiting Neural Vocoders and Audio Codecs for Voice Communication in Resource-Constrained Networks

A follow-up to my 2023 research and published paper on TTS and audio codec performance in resource-constrained networks. Was I right with my prediction?

In the second half of 2023, as part of my doctoral research at Montana State University, I conducted a study on text-to-speech (TTS) models and audio codecs for voice communication in bandwidth-constrained networks. The work compared traditional codecs, such as Codec2 at 3.2 kbps, against neural approaches including Meta’s Encodec, Google’s Lyra, and TTS + vocoder pipelines. Across 15 model-codec combinations, I evaluated Fréchet Distance, intelligibility, CLVP scores, and inference duration. The paper was later published at IEEE WiMob 2024.

The conclusions were cautiously optimistic. Neural vocoders and neural audio codecs showed clear advantages in quality-per-bit efficiency over traditional approaches, but inference speed on consumer hardware remained the main bottleneck. One TTS model paired with Style MelGAN, for example, could regenerate intelligible speech from minimal data, but required roughly 300 to 700 milliseconds per sample. That was too slow for real-time communication. As we wrote in the paper, “it is crucial to recognize that the inference process at the receiver does slightly increase latency compared to traditional audio codecs, resulting in delays between 1.2s to 1.9s.”

The broader implication was clear. Once inference caught up, whether through lighter models, better hardware, or both, neural codecs would become strong candidates for low-bandwidth voice communication.

Two years later, that prediction feels worth revisiting.

Over the past several weeks, I implemented, tested, and benchmarked a number of neural audio codecs, sometimes referred to today as audio tokenizers, with the assistance of AI tools. I wanted to see how much the landscape had changed, and whether real-time neural voice on consumer mobile devices had finally become practical.

This article is that follow-up.

The Goal: Real-Time Neural Audio on a Mobile Device

The goal was straightforward in theory: take a neural audio codec, run it in a development environment, port it to Android, optimize it, enable hardware acceleration, and ideally achieve real-time streaming encode and decode on a modest mobile hardware.

For voice communication, the requirement is not simply fast inference in batch mode. It is near-real-time, frame-by-frame processing with latency low enough for live conversation. Processing a full one-second clip in 250 milliseconds sounds impressive, but that is not the same as processing each 80 millisecond frame as it arrives. The distinction is architectural, not just computational.

That turned out to be the central challenge.

The issue was not that neural codecs performed poorly. In fact, many of them were surprisingly capable, even on fairly ordinary mobile hardware. The problem was that most were built for offline or batched processing. They could reconstruct audio faithfully, but they were not designed to operate as streaming systems.

What follows is a summary of what I learned by implementing several of them.

The Codecs

1. Stable Codec (Stability AI) | [GitHub]

Architecture: Transformer encoder-decoder with Finite Scalar Quantization (FSQ) Bitrate: 400 to 1000 bps (configurable) Sample rate: 16 kHz

Stable Codec was the first model I tried. It is a transformer-based audio autoencoder from Stability AI that uses sliding-window attention and FSQ for tokenization. On paper, it looked ideal. It has configurable bitrate down to 400 bps, strong quality metrics, and a relatively clean API through stable-audio-tools.

The problem appeared immediately. The model has a hard dependency on FlashAttention. It does not simply run more slowly without it. The project documentation explicitly states that inference quality and performance degrade substantially without FlashAttention. Since FlashAttention is CUDA-specific, that left no viable path to CPU or mobile inference.

That made Stable Codec a dead end for this use case.

Lesson: Check hardware requirements before writing a single line of integration code.

2. L3AC (Lightweight Lossless Audio Codec) | [GitHub]

Architecture: CNN encoder + local transformer + FSQ quantizer + CNN decoder Bitrate: 1 kbps Sample rate: 16 kHz Compression: 133x versus 16-bit PCM recording

I built a complete Rust implementation of L3AC using the Candle ML framework, including Metal GPU acceleration on macOS. The results were encouraging:

| Platform  | Encode (1s audio)  | Decode (1s audio)  | Realtime Factor  |
| - - - - - | - - - - - - - - - -| - - - - - - - - - -| - - - - - - - - -|
| CPU       | 127ms              | 343ms              | 2.1x             |
| Metal (M4)| 28ms               | 167ms              | 5.1x             |
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Quality was excellent, with a 0.9995 correlation against the Python reference implementation. The small divergence was most likely due to floating-point differences between Rust and Python.

However, L3AC processes audio in fixed windows rather than as a continuous stream. There is no mechanism to carry state across chunks. You provide a complete utterance, and it returns tokens. It is fast, accurate, and high quality, but not streaming.

3. LinaCodec | [GitHub] (fork of Kanade | [Github])

Architecture: WavLM for self-supervised features + Vocos vocoder + FSQ Bitrate: 170 bps Sample rate: 24 kHz input, 48 kHz decoded

LinaCodec was the most impressive codec I tested in terms of raw compression. It uses a distilled WavLM model to extract self-supervised speech features, quantizes them with FSQ, and reconstructs the waveform through a Vocos vocoder.

At roughly 170 bps (bits per second), the output was remarkably clear. In casual listening, it was close to indistinguishable from the original, although it did shift my accent and intonation somewhat. That is likely a consequence of being trained on LibriTTS.

This lines up closely with what my original research suggested. A neural vocoder pipeline can achieve extraordinary compression by representing speech through compact linguistic and prosodic features rather than trying to preserve the full waveform. The encoded tokens do not carry detailed acoustics in the traditional sense. Instead, they capture what is being said and some aspects of how it is said, while the vocoder regenerates plausible natural speech at the receiver.

On a Mac, LinaCodec ran at approximately 4x realtime. That is practical from a throughput perspective. The issue, again, was architectural. The model needs complete audio input. The WavLM encoder depends on full-context self-supervised features, and the global transformer requires access to the full sequence before producing final tokens. There is no streaming mode.

In that sense, LinaCodec is a strong validation of the TTS + vocoder thesis from my earlier work. It demonstrates that extremely low bitrate speech is possible when the system transmits a compressed intermediate representation and reconstructs natural-sounding audio at the receiver. Traditional codecs do not operate that way. Codec2, for example, can go as low as 700 bps, but the result is barely intelligible and clearly robotic. LinaCodec sounds dramatically better at a fraction of the bitrate.

Still, the model is inherently non-causal. Without major redesign, it is not suitable for real-time streaming voice.

4. HilCodec | [GitHub]

Architecture: SEANet encoder/decoder + Residual Vector Quantization (RVQ) + Short-time Fourier Transform (STFT) spectral processing Bitrate: 6 kbps (8 codebooks) Sample rate: 24 kHz

HilCodec uses a convolutional encoder-decoder similar to Encodec’s SEANet backbone, along with residual vector quantization and an additional spectral injection path through causal STFT. I implemented it in Rust/Candle with 8 RVQ codebooks of 1024 entries each.

Architecturally, it looked closer to what a streaming system might need. The convolutional backbone is causal, and RVQ itself can operate frame by frame. But in practice, the spectral processing path and the interaction between encoder and decoder made it difficult to separate into a clean streaming pipeline without substantial re-architecture.

It was also running at around 6 kbps, which was above my target bitrate.

5. UniCodec | [GitHub]

Architecture: Multi-domain codec with Mixture-of-Experts (MoE) and domain-adaptive codebook Bitrate: Variable Sample rate: Variable across speech, music, and general audio

UniCodec, published at ACL 2025, takes a more ambitious approach. It aims to handle speech, music, and general audio in a unified architecture using Mixture-of-Experts routing and domain-adaptive codebooks.

My main interest was speech compression, not general audio, but I implemented both a Python reference and a Rust version out of curiosity.

The research contribution is substantial. The unified architecture and self-supervised mask prediction ideas are genuinely interesting. But for the specific problem of real-time voice communication on mobile, the model felt overengineered. The MoE routing, multi-domain adaptation, and reliance on broader context made it impractical for streaming deployment.

6. Mimi (from Kyutai’s Moshi | [GitHub])

A useful reference point here is that NVIDIA’s PersonaPlex showed real-time neural voice capability with a papaer came out in January 2026, but Kyutai’s Moshi came almost a year earlier. Mimi is the codec used inside Moshi, a full-duplex voice AI assistant.

Mimi neural audio codec and its building blocks. Credits: Kyutai. [https://github.com/kyutai-labs/moshi]

This was the model that changed the direction of the project.

Unlike the other codecs, Mimi was designed from the start for real-time conversational streaming rather than offline batch reconstruction. That design choice matters more than any single metric.

Every major component in Mimi is causal. The SEANet backbone uses causal padding. The transformer uses causal attention with Rotary Positional Embeddings (RoPE). The split residual vector quantizer works on individual frames. In practical terms, that means you can feed the model 80 milliseconds of audio, obtain tokens, decode them back into waveform, and continue forward while preserving state between frames. That state is maintained through explicit convolutional buffers and a key-value (KV) cache for attention history.

This is why Mimi stood apart from the rest. It was not merely fast enough. It was architecturally aligned with the requirements of streaming voice communication.

Deep Dive: Porting Mimi to Android

Extracting from Moshi

Mimi is not packaged as a standalone project. It lives inside the kyutai-labs/moshi monorepo. Extracting it required pulling out the SEANet encoder and decoder, the transformer with RoPE attention, the streaming infrastructure, and the split residual vector quantizer, then rewiring them into an independent Rust crate.

The result was [mimi-core](https://github.com/tatujan/MimiCodec), roughly 3,500 lines of Rust across nine source files, built on the Candle ML framework. On a Mac M4, the Candle backend achieved 9.1x realtime encode and 11.1x realtime decode with Metal GPU acceleration.

The Mobile Performance Wall

On Android, there is no Metal backend. Candle falls back to CPU execution, and performance dropped to around 0.6x realtime. In other words, the codec was encoding more slowly than audio was arriving.

That is exactly the bottleneck my original study identified. Neural codecs can offer significantly better quality at lower bitrates, but inference speed on consumer devices determines whether they are actually deployable.

On a Mac with GPU acceleration, 9x realtime leaves plenty of margin. On a mid-range phone CPU, 0.6x is a non-starter.

ONNX Runtime and NNAPI

The breakthrough came from moving away from Candle’s CPU backend and switching to **ONNX Runtime, which can target [NNAPI](https://developer.android.com/ndk/guides/neuralnetworks)**, Android’s Neural Networks API for hardware-accelerated inference on supported devices.

That transition required a fairly involved export pipeline.

Mimi’s streaming architecture depends on three main state components:

• 11 convolutional state buffers per encoder/decoder, which carry the tail of causal padding between frames
• 16 KV cache tensors per encoder/decoder, corresponding to 8 transformer layers with key and value tensors
• a causal attention mask that prevents each frame from attending to future positions

The export script traces the PyTorch model into ONNX and makes all state tensors explicit inputs and outputs. At runtime, the Rust Java Native Interface (JNI) bridge manages those tensors frame by frame, feeding updated convolutional states and KV caches back into the model on each step.

One useful detail that emerged during development was that chunk size is not fixed in the ONNX graph. Because the exported model uses dynamic axes on the audio input and KV cache dimensions, the runtime can accept any multiple of 80 milliseconds. In the [MimiDemo Android app](https://github.com/tatujan/MimiDemo/tree/feature/onnx-backend), users can select chunk sizes directly from the UI. Larger chunks reduce the number of ONNX Runtime calls and improve throughput, but they do so at the cost of higher latency.

I also explored an additional optimization around model storage via weight-only FP16 conversion. In practice, this did not improve runtime performance, because NNAPI was already handling precision optimization internally and executing efficiently in FP16 where appropriate. But it did make a meaningful difference in model size, which matters on mobile.

The key distinction is that NNAPI can optimize execution at runtime, while the model files themselves still store full FP32 weights unless they are explicitly converted. To reduce that footprint, I created weight-only FP16 variants of the streaming ONNX models. A conversion script rewrites FP32 initializers, meaning the learned weights, as float16 and inserts Cast nodes so computation remains unchanged during inference. In other words, the model runs the same way and preserves the same output quality, but occupies substantially less space on disk. The reduction was significant, more than 40%.

Results

With ONNX Runtime and NNAPI acceleration:

| Metric                  |  Value        |
| - - - - - - - - - - - - |- - - - - - - -|
| Encode speed            | 2-4x realtime |
| Decode speed            | 2-4x realtime |
| Bitrate 8 CB            | 1.1 kbps      |
| Bitrate 16 CB           | 2.2 kbps      | 
| Frame latency           | 80ms          | 
| Model size (8 CB) (FP32)| 364 MB        |
| Model size (8 CB) (FP16)| 212 MB        |
- - - - - - - - - - - - - - - - - - - - - -
* CB: Codebook

A 2x realtime factor means the codec processes audio twice as fast as it arrives, which is enough headroom for a real-time voice pipeline with additional room for network overhead and application logic.

What This Proves

My original research identified the main gap clearly: neural methods offered better quality-per-bit than traditional codecs, but they were not fast enough on average consumer hardware to support real-time use.

That picture has changed.

First, the compression thesis is clearly validated. LinaCodec reaches approximately 170 bps while still producing regenerated speech that sounds impressively natural. That is the neural vocoder idea taken to its logical conclusion: encode the linguistic structure and let the receiver regenerate the acoustics. My earlier work pointed toward that direction when comparing TTS + vocoder systems against conventional codecs.

Second, the inference gap is no longer hypothetical. Mimi runs at around 2x realtime on an ordinary Android phone. Not a premium flagship device, but a mid-range phone. Once hardware acceleration through NNAPI is combined with a codec that was designed for streaming from the beginning, real-time neural voice becomes practical.

Third, streaming is the real constraint. Many of the codecs I implemented produced excellent results in batch mode. Only one of them, Mimi, was architected for frame-by-frame streaming, with its causal architecture. That is not something that can easily be added later. It has to be built into the model from the start.

What Comes Next

This field is moving quickly. New neural codec architectures are appearing every few months, hardware-accelerated inference keeps improving, and the distance between research prototypes and deployable systems is disappearing.

At this point, the question is no longer whether neural codecs will eventually displace traditional codecs for low-bandwidth voice communication. The more interesting question is which architecture will do it first, and which tradeoffs will prove acceptable in real-world systems.

That part is still open.