Why Cross-Attention Breaks in Diffusion-Based TTS — And Three Ways to Fix It

A deep dive into phoneme alignment, the early-timestep failure problem, and LARoPE

Speech Synthesis · Deep Learning · Positional Encoding · 2026

Building a text-to-speech system with a diffusion model introduces a problem that doesn’t get talked about enough: cross-attention for phoneme alignment is fundamentally broken at early diffusion timesteps.

This isn’t a minor implementation detail. It’s a structural contradiction built into the architecture — and different systems have arrived at very different solutions. This post walks through the problem, why it matters, and how F5-TTS, SupertonicTTS, and VoxFlash-TTS each handle it.

The Problem: What Happens to Cross-Attention at Early Timesteps

In a diffusion-based TTS system, you need to map a variable-length phoneme sequence onto a variable-length audio sequence. Cross-attention is the natural tool: audio frames as queries, phoneme embeddings as keys and values.

But here’s the structural issue. The diffusion forward process is:

z_t = √ᾱ_t · z_0 + √(1 - ᾱ_t) · ε

where z_0 is the clean audio latent, ε is Gaussian noise, and ᾱ_t decreases monotonically with timestep t.

At early timesteps (t close to T, ᾱ_t ≈ 0):

z_t ≈ ε    (almost pure noise, no audio structure)

At late timesteps (t close to 0, ᾱ_t ≈ 1):

z_t ≈ z_0  (audio structure restored)

Now think about what happens to cross-attention at early timesteps. The query vectors come from z_t — which is essentially random Gaussian noise. These noise vectors have no semantic relationship to the phoneme embeddings they're supposed to attend to.

The attention weight computation:

Attention(Q, K, V) = softmax(QK^T / √d) · V

When Q is noise, the dot products QK^T are essentially random. The resulting attention weights tend toward a uniform distribution — the model has no idea which phoneme each audio frame should correspond to.

The core contradiction: alignment requires meaningful queries, but early-timestep queries are noise.

This problem is specific to the combination of non-autoregressive diffusion and cross-attention alignment. Autoregressive systems (VALL-E) don’t have it because each generation step has explicit context. FastSpeech-style systems don’t have it because they use explicit duration prediction and never rely on cross-attention for alignment.

Three Solutions

Solution 1: F5-TTS — Avoid Cross-Attention Entirely

The idea: Don’t use cross-attention for alignment. Encode the alignment implicitly through sequence construction.

F5-TTS concatenates the phoneme sequence with the noisy audio sequence before the Diffusion Transformer (DiT):

Step 1: Pad phoneme sequence with filler tokens to match audio length
        [ph_1, ph_2, ph_3, filler, filler, ...]   shape: [T_audio, d_text]

Step 2: Concatenate on the feature (channel) dimension — not sequence dimension
        [ph_1, ph_2, ph_3, filler, ...]            shape: [T_audio, d_text]
                  ↕ concat on feature dim
        [z_t_1, z_t_2, z_t_3, ...]                 shape: [T_audio, d_audio]
                  ↓
        Output: sequence length stays T_audio, feature dim expands to d_text + d_audio

Step 3: Process with self-attention + RoPE inside DiT blocks

Why this works: Once you concatenate, alignment is encoded structurally. A phoneme at position 0.3 in the sequence is physically adjacent to the audio frame at position 0.3. Self-attention with RoPE can exploit this positional relationship without needing the query to carry semantic meaning — the position itself is the signal.

The early-timestep problem disappears because there’s no cross-attention to fail. Self-attention operates on the concatenated sequence, and RoPE encodes relative positions regardless of whether the audio portion is noisy or clean.

Trade-offs: The concat happens on the feature dimension, so sequence length stays at T_audio — no doubling. However, each position’s feature dimension increases (d_text + d_audio), which raises per-token compute. The model needs to learn to ignore filler tokens. The alignment is implicit — requiring more model capacity and training data to learn reliably. RoPE here is doing its standard job in self-attention, not specifically designed for text-speech alignment.

Solution 2: SupertonicTTS — LARoPE

The idea: Keep cross-attention, but fix the positional encoding so that position itself provides an alignment signal when semantic content is absent.

SupertonicTTS adopts LARoPE (Length-Aware Rotary Position Embedding), introduced in Kim et al. (2025, arXiv:2509.11084).

The problem with standard RoPE in cross-attention

Standard RoPE encodes positions as absolute integer indices:

Text sequence:  position 0, 1, 2, ..., T_text
Audio sequence: position 0, 1, 2, ..., T_audio

The two sequences have different lengths. Position 5 in a 10-token text sequence is the midpoint; position 5 in a 200-frame audio sequence is near the start. The indices aren’t comparable — cross-attention can’t use position to infer alignment.

When the query is noise (early timesteps) and position doesn’t help, alignment has nothing to anchor to.

What LARoPE does

LARoPE normalizes positions relative to each sequence’s total length:

Text position i   →  normalized: i / T_text   ∈ [0, 1]
Audio position j  →  normalized: j / T_audio  ∈ [0, 1]

Now both sequences live in the same positional space. Audio frame at normalized position 0.3 will have high rotary similarity with text tokens near normalized position 0.3 — regardless of what the query vector contains semantically.

The behavior across timesteps:

Early timesteps (Q ≈ noise):
  Attention weight ≈ f(positional similarity)
  → LARoPE provides a diagonal prior: each audio frame
    attends to roughly the corresponding text region
  → Approximate monotonic alignment emerges from position alone

Late timesteps (Q recovering audio structure):
  Attention weight ≈ f(semantic similarity + positional similarity)
  → Semantic content increasingly drives alignment
  → Fine-grained correspondence refines

This is an elegant progressive mechanism: position anchors alignment in the early timesteps, then steps back as semantic content takes over. The transition is smooth and requires no extra modules.

The diagonal property: The authors show that LARoPE consistently preserves a diagonal structure in the relative upper bound matrix regardless of sequence length — directly analogous to the natural monotonic alignment between text and speech. Standard RoPE loses this diagonal structure when sequence lengths differ significantly.

Experimental results: LARoPE achieves state-of-the-art WER among zero-shot TTS models that rely on attention for alignment, with faster loss convergence and greater robustness to variable utterance duration (up to 30 seconds, where standard RoPE degrades notably).

SupertonicTTS also uses Context-sharing Batch Expansion alongside LARoPE — a training strategy that accelerates alignment convergence by sharing context across batch items. LARoPE provides the positional prior; Context-sharing Batch Expansion accelerates the learning of semantic alignment on top of it.

Solution 3: VoxFlash-TTS — Move Alignment Outside the Diffusion Model

The idea: Don’t solve the early-timestep problem. Eliminate it by resolving alignment before diffusion begins.

VoxFlash-TTS uses a coarse-grained explicit alignment approach:

Before diffusion:
  Text → Phoneme sequence
       ↓
  External duration prediction tool
       ↓
  Per-phoneme duration (number of frames)  ← alignment computed here
       ↓
  Expand phoneme sequence to match audio length

During diffusion:
  Model receives pre-aligned conditioning — no alignment learning required
  Denoising proceeds on already-aligned inputs

The diffusion model never needs to figure out which audio frame corresponds to which phoneme. That mapping was determined externally, before the first diffusion step. The noisy query problem simply doesn’t arise.

Trade-offs: This introduces a dependency on an external alignment tool. The alignment granularity is coarse — fine-grained rhythmic nuance may be lost. If the duration prediction is wrong, the diffusion model can’t correct it. The pipeline is more complex.

But for inference speed, this approach has a significant advantage: the diffusion model receives clean, pre-structured conditioning, and with VoxFlash’s 9 Hz latent space compression (reducing the sequence from ~750 to 90 vectors), the denoising computation is minimal.

The Deeper Pattern

Stepping back, all three solutions share a common structure: they compensate for the absence of semantic signal in early-timestep queries with some form of prior.

SystemPrior typeStrengthFlexibilityF5-TTSStructural (feature-dim Concat)Implicit, weakHighSupertonicTTSPositional (LARoPE normalization)Explicit, mediumHighVoxFlash-TTSExternal (duration prediction)Hard constraintLow

F5-TTS encodes the weakest prior — position is implied by concatenation order, but the model still has to learn to use it. SupertonicTTS encodes a medium-strength prior — LARoPE explicitly biases attention toward position-matched pairs, but lets the model deviate when semantics demand it. VoxFlash-TTS encodes the strongest prior — alignment is fixed before generation, leaving no room for the model to learn or adapt.

Stronger priors trade flexibility for stability. Which is the right trade-off depends on what you’re optimizing for.

What This Means for System Design

If you’re building or selecting a TTS system and alignment quality matters:

If you want maximum alignment quality with minimal architectural complexity: F5-TTS’s Concat approach is surprisingly effective despite its simplicity. The implicit positional prior from concatenation, combined with a capable DiT, handles most practical cases well.

If you want cross-attention flexibility with robust alignment: LARoPE is the most principled solution to the early-timestep problem specifically. It’s a targeted fix that improves alignment without restructuring the architecture — a simple extension of RoPE that directly addresses the root cause.

If you want alignment to be a non-issue and can tolerate external dependencies: Explicit pre-alignment removes the problem entirely at the cost of pipeline complexity and alignment flexibility.

Summary

The early-timestep cross-attention failure in diffusion TTS is a real structural problem, not a quirk. At early timesteps, query vectors are noise — they carry no phoneme correspondence information. Cross-attention without a compensating mechanism produces random alignment weights.

Three systems address this differently:

• F5-TTS removes cross-attention from the alignment path, using self-attention on a concatenated sequence where position encodes alignment implicitly
• SupertonicTTS uses LARoPE to normalize text and audio positions to the same scale, so positional similarity anchors alignment when semantic content is absent
• VoxFlash-TTS resolves alignment externally before diffusion begins, making the early-timestep problem irrelevant

LARoPE is the most direct solution to the stated problem — it targets exactly the mechanism that fails and fixes it with a lightweight modification. The diagonal positional prior it creates maps naturally onto the monotonic structure of text-speech alignment, providing a stable foundation for the semantic alignment that develops at later timesteps.

References

• LARoPE: Kim et al. (2025). Length-Aware Rotary Position Embedding for Text-Speech Alignment. arXiv:2509.11084
• SupertonicTTS: Kim et al. (2025). SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech System. arXiv:2503.23108
• F5-TTS: Chen et al. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv:2410.06885
• VoxFlash-TTS Demo: voxflash.github.io
• VoxFlash-TTS GitHub: github.com/VoxFlash/VoxFlashTTS