How Transformer Processes Audio

Everything you need to understand how modern speech AI actually works. Sound is just numbers. Here’s how transformers turn those numbers into meaning.

You say, “Hey Siri.” Before you’ve finished the second syllable, something extraordinary has already happened. A model somewhere took the pressure waves coming out of your mouth, the physical disturbance of air molecules, and turned them into meaning. Into intent. Into a command it can act on.

This post is about this gap, the gap between a sound wave and a word. Between vibrating air and a transformer, paying attention. We’re going to close that gap completely; from the physics of sound, through the machinery of spectrograms, all the way to the architecture decisions that make models like Whisper, Wav2Vec2, and SpeechT5 work the way they do.

Sound Is Just Numbers

Here’s the first thing you need to internalize: sound is analog. It’s a continuous wave of air pressure, rising and falling thousands of times per second. Your eardrum literally moves back and forth as these pressure changes hit it. That continuous motion is what your brain eventually decodes as speech, music, or a fire alarm.

Computers can’t store continuous things. They store lists of numbers. Discrete numbers. So the first problem in audio AI is: how do you turn an infinite, continuous wave into a finite list of numbers without losing what matters?

The answer is sampling. Imagine the sound wave as a curve drawn on paper. You can’t store the entire curve, but you can take measurements, “at this moment in time, the pressure(amplitude, for instance) is this high”, and store those measurements. If you take enough measurements, closely enough together, the list of numbers becomes a faithful approximation of the original curve.

The rate at which you take these measurements is called the sampling rate, measured in Hz (samples per second).

A few numbers to anchor your intuition:

• 8000 Hz: old telephone quality sound. 8000 snapshots or samples per second. Sounds muffled, like someone talking through a wall.
• 16000 Hz: standard for speech AI models (ASR, STT, TTS, etc.). Enough to capture everything meaningful in human speech. Mostly used.
• 44100 Hz: CD quality sound. Overkill for speech, appropriate for music.

Why is 16,000 Hz enough for speech? Because of the **Nyquist theorem, **a beautiful piece of information theory that says: you only need to sample at twice the highest frequency you care about to perfectly reconstruct it. Human speech lives below 8,000 Hz. Sample at 16,000 Hz. Done. You’ve captured everything your model needs.

Think of it this way. Imagine a wave that completes one full cycle, up and down, every millisecond. To know that a wave existed, you need to catch it at least twice per cycle: once on the way up, once on the way down. Miss either measurement, and you lose the wave entirely — it slips through the gaps between your snapshots like water through open fingers. That’s all Nyquist is saying. Two samples per cycle, minimum. Human speech cycles at most 8,000 times per second, so you need at least 16,000 samples per second to catch every cycle. Sample faster, and you’re just taking redundant measurements of something you already captured perfectly. Sample slower and some cycles slip through, and your model will never know they were there.

Go lower, and you lose the crispness of consonants. The difference between “ship” and “chip” lives in those high-frequency transients. Lose them, and your model can’t tell the words apart either.

Each sample also has a precision, how accurately you record the pressure value at that instant. This is called bit depth. 16-bit audio gives you 65,536 possible values. 24-bit gives you 16 million. In practice, 16-bit is already more precise than your ears can detect the error.

One more thing: before audio goes into any machine learning model, every sample value gets converted to a float between -1.0 and 1.0. Models speak floating point. This is non-negotiable, and every audio library does it automatically.

So now you have your audio as a long list of floats. A 5-second recording at 16,000 Hz is a Python list with 80,000 numbers in it. That’s your raw material.

The question is: what do you do with it?

The Spectrogram: Seeing Sound

Here’s the problem with raw waveforms. They contain everything, but in a form that’s almost impossible to learn from directly.

Look at a raw waveform, and you see a wiggly line. Is that person saying “hello” or “yellow”? You can’t tell. The information is there, but it’s buried inside rapid oscillations at different frequencies, all superimposed on top of each other. It’s like trying to read a book where every page is printed on top of every other page simultaneously.

What you need is a way to separate the frequencies. To look at a sound and say: “Right now, from 0.0 to 0.025 seconds, there’s a lot of energy at 500 Hz, some energy at 2000 Hz, and very little above 4000 Hz.” That description is infinitely more useful than the raw wiggle.

The tool or algorithm that does this is the Short-Time Fourier Transform (STFT).

The idea is simple, even if the math underneath is not: chop the audio into tiny overlapping windows (say, 25 milliseconds each). For each window, run a Fourier transform, a mathematical operation that decomposes any signal into its constituent frequencies and tells you the strength of each. Stack all these frequency snapshots side by side, and you get a spectrogram.

A spectrogram has a shape: (frequency bins × time steps). It’s a 2D matrix. And here’s the beautiful accident that changed everything in audio AI: it looks like an image.

The x-axis is time, the y-axis is frequency, and the pixel brightness is energy. Vowels look like bright horizontal bands. Consonants look like vertical bursts. Silence looks like black. Music looks like a painting.

But we can go one step further. The raw spectrogram uses a linear frequency scale, which means equal spacing between 100 Hz and 200 Hz, as between 1000 Hz and 1100 Hz. Human ears don’t work that way. We’re much more sensitive to differences at low frequencies than high ones. The difference between 200 Hz and 400 Hz is dramatic to us; the difference between 5000 Hz and 5200 Hz is barely noticeable.

So we compress the frequency axis onto the mel scale, a scale that matches human perceptual sensitivity. We also take the log of the energy values (expressing them in decibels) because loudness perception is also logarithmic.

The result is a log-mel spectrogram. It’s what you feed to almost every serious audio model today. It’s the canonical input format because it captures what matters to a human listener and discards what doesn’t.

One important caveat before we move on: spectrograms and images look identical, but they’re not the same kind of object. If you take an image of a cat and shift it 50 pixels to the right, it’s still a cat, same meaning. If you take a spectrogram and shift it upward, moving all the frequency energy higher, you’ve fundamentally changed the sound. Different pitch. Potentially a completely different phoneme. Spectrograms are not translation-invariant(which change meaning on translation), the way images are. Models that treat them as images can work remarkably well in practice, but this difference is real and worth remembering when things go wrong.

What a Transformer Actually Receives

Now we’re at the bridge. You have audio. You have a spectrogram. How does any of this become input to a transformer?

Transformers process sequences of vectors. Give them a sequence of vectors, they’ll do attention over it and give you back another sequence of vectors.

The entire challenge of audio AI is: how do you turn sound into a sequence of vectors that a transformer can meaningfully attend over?

There are two main approaches, and the models you’ll use every day make different choices here.

Path 1: The Raw Waveform Approach (Wav2Vec2, HuBERT)

Some models take the raw waveform directly, that list of 80,000 floats, and process it through a small convolutional neural network first. The CNN acts as a feature extractor: it slides over the raw audio samples, compresses them, and produces one embedding vector per approximately 20 milliseconds of audio.

So 1 second of audio at 16,000 Hz becomes 50 vectors. Each vector is 512-dimensional and represents what was happening in that 20ms window of audio. These 50 vectors then go into the transformer as a sequence.

Path 2: The Spectrogram Approach (Whisper, AST)

Other models first convert the audio to a log-mel spectrogram and then feed that spectrogram to the transformer. Whisper, for example, converts audio to an 80-channel log-mel spectrogram and then processes it through two convolutional layers before passing it into the transformer encoder.

The **Audio Spectrogram Transformer (AST) goes even further and treats the spectrogram as a literal image: it cuts it into 16×16 pixel patches (just like [ViT](https://arxiv.org/abs/2010.11929)** does with images), projects each patch into an embedding vector, and feeds the sequence of patch embeddings to a standard vision transformer.

Both paths produce the same thing: a sequence of fixed-size vectors. And once the transformer has that sequence, it doesn’t care whether it came from raw audio or a spectrogram. It just does attention.

This is the key insight: the transformer architecture is modality-agnostic. What changes between an NLP transformer and an audio transformer is almost entirely in the input preprocessing, like how you convert your raw data into a sequence of vectors. The attention mechanism itself is identical.

If you’ve spent time with BERT or GPT, you’re already 80% of the way to understanding audio transformers. The remaining 20% is everything we’ve discussed so far about spectrograms, waveform encoders and all these input pre-processings.

Three Heads, Three Tasks

Now things get interesting. Once you have an encoder producing a sequence of hidden states, one per 20ms of audio, what you do with those hidden states determines everything. The encoder is a feature extractor. The head you put on top of it determines what task you’re solving.

There are three fundamental patterns.

1. Teaching the Model to Spell From Scratch

Connectionist Temporal Classification(CTC) is the approach used by Wav2Vec2, HuBERT, and similar encoder-only models for speech recognition.

Here’s the fundamental problem it solves. When you train a speech recognition model, your dataset is pairs of (audio, transcript). Nobody told you when each word occurs in the audio. The dataset just says, “This 4-second clip contains the words: Alice is reading a blog post on audio AI” Where does “Alice” end and “reading” begin? No idea.

This is the alignment problem. CTC solves it elegantly.

The model predicts one character every 20ms of audio. For 4 seconds of audio, that’s 200 character predictions. But “Alice is reading a blog post on audio AI” is only 39 characters. What happens to the other 161 time steps?

The model is allowed to output a special blank token, written as _. Think of it as the model saying, "I'm not sure what character this is, or I'm mid-character right now." The raw output for the word "ERROR" might look like this:

_ER_RRR_ORR

That’s not a bug. It’s the model correctly expressing the temporal smearing of speech, the letter R in “error” is held for longer than 20ms, so it gets predicted multiple times. The blank token _ acts as a hard boundary between character groups. Decoding this back to text uses two simple rules:

1. Collapse consecutive duplicates within each group: **RRR → R, RR → R**
2. Remove all blank tokens

So _ER_RRR_ORR becomes _ER_R_OR, and then remove all the blank tokens, it becomes ERROR.

What CTC loses: it only sees individual characters, never whole words. It can output phonetically plausible but wrongly spelled words. This is why CTC models are often paired with an external language model that acts as a spellchecker on top.

What CTC wins: it’s fast. The encoder runs once and produces all 200 predictions in a single forward pass. No loop. No sequential generation. For real-time applications where latency matters, this is significant.

2. The Full Encoder-Decoder Architecture

This is the Whisper approach, and it’s more powerful. The encoder does its job, reads the audio, and produces a sequence of rich hidden states that represent “what is being said.” Then the encoder hands off to a decoder.

The decoder is an autoregressive language model. It generates text one token at a time, in a loop:

At every step, the decoder looks at two things: the tokens it’s already generated (via self-attention) and the encoder’s audio representation (via cross-attention). The cross-attention is the bridge; it’s how the decoder continuously consults the audio while generating each word.

Two important differences from CTC: First, Whisper uses a 50,000+ token vocabulary, full words and subwords, not individual characters. So instead of predicting 200 characters for a 4-second clip, it might predict 8 word tokens. The output is shorter, richer, and inherently correct in spelling because tokens come from a real vocabulary.

Second, because the decoder is a language model, it has strong priors about what words follow other words. It doesn’t need an external language model. The language model is already built in.

The cost: it’s slower. Every token requires a full decoder forward pass. For a long transcription, that’s many sequential steps that can’t be parallelized.

3. Classification Tasks

Sometimes you don’t need text at all. You just need a label. “Is this audio a lion roaring or a cat meowing?” “What music genre is this?” “Which speaker is talking right now?”

For these tasks, you take the encoder’s sequence of hidden states and do one of two things:

i. Sequence classification (one label for the whole clip): average all the hidden states into a single vector. Pass that vector through a linear layer. Get a probability distribution over your classes. Done. Simple.

ii. Frame classification (one label per 20ms): run the linear layer on every hidden state individually. Get a sequence of label probabilities. This is how speaker diarization works: the model outputs a different speaker label for each 20ms window, and you can track exactly when speakers change.

No decoder needed. No sequential generation. One forward pass through the encoder, one pass through the classification head, and output your answer.

Teaching Machines to Speak: the Reverse

Everything we’ve discussed so far is about going from audio to text or labels. But the architecture flips beautifully for the opposite direction: generating speech from text.

A Text-to-Speech(TTS) model is a seq2seq model running in reverse. The encoder reads text tokens and builds a representation of “what should be said.” The decoder then generates one time-slice at a time, a spectrogram.

There’s a subtlety here worth pausing on. The STFT that produces a spectrogram captures two things: amplitude (how loud each frequency is) and phase (the timing offsets of each frequency). Both are mathematically required to reconstruct the original waveform.

TTS decoders only predict amplitude. Phase is discarded during training because it’s extraordinarily difficult to predict and doesn’t affect perceptual quality much. This means you can’t simply run the inverse STFT to get your audio back; you’re missing half the information.

That’s what the vocoder is for. A vocoder (models like HiFi-GAN are common) is a separately trained neural network whose only job is: given an amplitude spectrogram, estimate the missing phase and produce a waveform. It’s learned from thousands of hours of real audio, so it knows what a plausible phase looks like. The resulting audio sounds natural because the estimated phase is close enough to reality that human ears can’t detect the error.

One more wrinkle that makes TTS fundamentally harder than ASR: it’s a one-to-many problem. For a given audio clip, there is exactly one correct transcript. But for a given piece of text, there are infinitely many valid ways to say it, different emphasis, different speed, different intonation, different emotion. A model that generates any of them is technically correct.

This is why you can’t evaluate TTS with a simple accuracy metric. Instead, researchers use MOS, Mean Opinion Score, where real humans listen to samples and rate them from 1 to 5. It’s slow, expensive, and the only evaluation that actually tells you whether the output sounds like a real person.

Conclusion

Everything in audio AI starts the same way: convert sound into a sequence of vectors (either through a CNN over raw waveforms, or through a spectrogram pipeline). Feed that sequence to a transformer encoder. Get back a sequence of rich hidden representations.

What you do next defines the task:

• Attach a CTC head: fast character-by-character speech recognition
• Attach an autoregressive decoder: powerful word-level transcription or speech synthesis
• Attach a classification layer: genre detection, speaker identification, emotion recognition

The encoder is the expensive, powerful, universal part. It’s what you pretrain on, hundreds of thousands of hours of audio. The head is almost an afterthought, a small linear layer that costs almost nothing to swap out.

This is why **transfer learning** works so well in audio AI: you train a great encoder once, then fine-tune it on your specific task with a small head. The encoder already knows what audio sounds like. You just teach it what to do with that knowledge.

Will cover TTS and ASR separately in detail. With practical code. Stay tuned.

Further readings: https://huggingface.co/learn/audio-course/chapter0/introduction https://arxiv.org/abs/2105.00335 — Audio Transformers https://arxiv.org/abs/2303.11607 — Transformers in Speech Processing: A Survey https://arxiv.org/abs/2212.04356 — Robust Speech Recognition via Large-Scale Weak Supervision https://arxiv.org/abs/2503.10446 — Whisper Speaker Identification: Leveraging Pre-Trained Multilingual Transformers for Robust Speaker Embeddings https://arxiv.org/abs/2006.11477 — wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations