Are audio codecs about context? #paper_reading

Codec: Codec Does Matter: Exploring the Semantic Shortcomings of Codecs for ALM

*Paper [Code](https://github.com/zhenye234/xcodec?tab=readme-ov-file)*

The trend of aligning different modalities is not new. One of the approaches is to put all responsibility on LLM (Large Language Model) and hope it will somehow understand the other modality. All that’s required from us is to transform our input modality into tokens.

If we dive deeper, we’ve always been doing this — aren’t those hidden representations from any SST models essentially tokens? The only difference now is that we need a number, not a vector. This is where codecs come in — about Encodec, you can read more here (link).

https://arxiv.org/abs/2408.17175

We can think of them as quantizers, converting m audio samples into n points. In essence, it’s similar to any other transformation. Initially, as I understand it, codecs were mainly used for music and sounds. As a result, we learned to capture low-level acoustic fluctuations. However, for end-to-end systems, we need to understand the content.

That’s why semantic tokens — tokens that carry semantic meaning — are introduced. Where do we learn semantics? In SST (Self-Supervised Training), which we later transfer to ASR (Automatic Speech Recognition) — this includes models like HuBERT and Wav2Vec.

Approach 1: We perform distillation on the first, second, or n-th layers. In SpeechTokenizer, this is cosine loss between the first RVQ (Residual Vector Quantization) layer and HuBERT features. This works due to the cascading nature of RVQ. A similar approach is proposed in Moshi.

Approach 2: Explicitly add semantic features — namely, before RVQ. Of course, you could just take HuBERT and the audio encoder features, but that’s not very fancy 🎩. So, we create a separate Encoder-Decoder that takes HuBERT features as input, then feeds them into a semantic encoder (using convolutions). Here, we capture the audio encoder features, concatenate them with the linear projection, and only then feed them into RVQ. What’s interesting is that we didn’t create an encoder-decoder for semantic features for nothing! We feed the RVQ output into the semantic decoder to recover our original HuBERT features using MSE loss. This is where the irony I mentioned earlier actually fails — the encoder-decoder is necessary to find the “correct” semantic features, which help recover both acoustics and retain HuBERT-level semantics. And they’re “correct” because they go through the RVQ, the very component we are working so hard on.

What approach do you like more?