Why Reference Audio Gets Contaminated by Noise in Masked Diffusion TTS

The structural problem with bidirectional self-attention in voice cloning

Speech Synthesis · Diffusion Models · Voice Cloning · 2026

Masked diffusion is one of the most architecturally elegant approaches to voice cloning. F5-TTS and VoiceBox both use it. The idea is simple: frame voice cloning as a speech infilling task, concatenate the reference audio and the audio to be generated along the sequence dimension, and let the model fill in the masked region with the target speaker’s voice.

But there’s a structural problem built into this design that doesn’t get talked about much: in DiT’s global bidirectional self-attention, reference audio frames attend to the noisy generation region — and the noise bleeds back into the reference representations, degrading the quality of voice timbre extraction.

Before going further, an important distinction: two interactions happen simultaneously in self-attention, and they’re fundamentally different in nature:

• Generation region attending to reference region → the model extracts timbre from the reference. This is intended and desirable.
• Reference region attending to generation region (noise) → noise contaminates the reference representations. This is harmful.

When we talk about “contamination” here, we mean only the second direction — the harmful one. The first direction is exactly what we want.

How Masked Diffusion Works

A quick recap of the architecture, using F5-TTS as the example.

The model receives three inputs concatenated along the feature dimension:

Sequence dimension:
[ref_frame_1, ref_frame_2, ... | gen_frame_1, gen_frame_2, ...]
 ←── reference region ──────→   ←── generation region ──────→

Feature dimension at each position:
  noisy speech   +   masked speech   +   text condition

• Noisy speech: the full sequence (reference + generation) with Gaussian noise added
• Masked speech: reference region retains the real mel spectrogram; generation region is zeroed out
• Text condition: phoneme sequence padded to sequence length

The model is trained on a speech infilling objective: given the reference region’s real mel spectrogram, generate the masked region with matching voice timbre.

The Reference Audio Contamination Problem

Self-attention is fully bidirectional

DiT blocks use global bidirectional self-attention — every position attends to every other position, with no causal mask:

Self-attention matrix:

                  reference region    generation region
reference region  [  ✅ ↔ ✅        |  ✅ ← noise  ]  ← contamination (harmful)
generation region [  ✅ → ref       |  ✅ ↔ ✅      ]  ← timbre extraction (intended)

The reference region attending to the noisy generation region is the problem. Reference frame representations get mixed with noise information they shouldn’t contain.

Contamination accumulates across layers

In a multi-layer DiT, the noise contamination of reference representations compounds:

Layer 1:
  Input:  reference (clean)  ↔  generation (noise)
  Output: reference_repr_1 (lightly noise-contaminated)

Layer 2:
  Input:  reference_repr_1 (lightly contaminated)  ↔  generation_repr_1
  Output: reference_repr_2 (more noise-contaminated)

...

Layer N:
  Input:  reference_repr_{N-1} (heavily contaminated)
  Final:  generation region extracts timbre from heavily noise-contaminated reference

By the time the generation region attends to the reference in deep layers, it’s not attending to the original reference audio features — it’s attending to a representation that has been progressively mixed with noise through repeated attention.

Contamination is worst at early timesteps

The severity is directly tied to the diffusion timestep:

Early timesteps (t close to T, generation ≈ pure noise):
  Reference contamination: severe
  → Generation region extracts timbre from heavily contaminated reference
  → Timbre extraction quality: poor

Late timesteps (t close to 0, generation ≈ clean audio):
  Reference contamination: mild
  → Timbre extraction quality: improving

This timing matters. Early diffusion timesteps are when the model makes the most consequential structural decisions. Poor timbre extraction at this stage means the generation trajectory starts misaligned with the target voice — and later steps have limited ability to correct it.

The Masked Speech Anchor: Partial Mitigation

F5-TTS’s design includes a feature that partially addresses this: the masked speech input.

At every timestep, the model receives the clean reference mel spectrogram as a separate channel alongside the noisy speech:

At each DiT layer, each reference position receives:
  - Noisy speech channel: noise-contaminated (participates in bidirectional attention)
  - Masked speech channel: clean real mel spectrogram (always present via channel concat)

The clean masked speech acts as a persistent anchor — a noise-free signal present at every layer through the feature dimension, counteracting the noise bleeding in from bidirectional attention.

However, this mitigation is incomplete:

• The masked speech is injected via channel concatenation, not by replacing the hidden state
• Bidirectional self-attention still mixes noise into the reference region’s hidden representation
• With enough layers, the noise influence accumulates in the hidden states regardless of the clean anchor

Theoretical Solutions

1. One-directional attention mask

Block reference frames from attending to generation frames:

Masked attention:

                  reference region    generation region
reference region  [  ✅ ↔ ✅        |  ❌ (blocked)   ]  ← noise contamination prevented
generation region [  ✅ → ref       |  ✅ ↔ ✅         ]  ← timbre extraction preserved

Reference representations are determined only by other reference frames. Generation frames can still attend to clean reference representations and extract timbre normally.

The cost: reference frames can no longer sense the generation region content, which may limit some global modeling capabilities. It also requires modifying the DiT attention implementation.

2. Cross-attention instead of self-attention

Encode the reference independently, then let generation frames query the reference encoding via cross-attention. The two regions never share a self-attention sequence — reference noise contamination is eliminated at the architectural level.

The cost: loses the simplicity advantage of masked diffusion; effectively becomes a different injection method.

3. Curriculum learning on noise level

During training, start with low-noise generation regions and gradually increase intensity. The model develops robust timbre extraction before being exposed to heavy noise contamination.

How Serious Is This in Practice?

F5-TTS achieves strong zero-shot voice cloning results despite this structural issue. The contamination problem is real, but it doesn’t prevent the approach from working.

Evidence that it matters:

• Reference audio selection has outsized impact on cloning quality in F5-TTS — longer references don’t always help and can sometimes hurt, consistent with noise contamination accumulating over longer sequences
• Early-timestep timbre extraction quality affects the generation trajectory in ways that later steps can only partially correct

Evidence that the impact is limited:

• The masked speech anchor provides a meaningful clean signal at every layer
• Large-scale training gives the model enough exposure to learn noise-contamination-robust representations
• F5-TTS demonstrates that masked diffusion works well in practice despite this theoretical limitation

The honest conclusion: reference audio noise contamination is a real structural limitation — but the masked speech anchor and large-scale training mitigate it enough that the approach remains practically effective.

Comparison with Other Injection Methods

| Method | Reference in self-attention? | Reference noise contamination | Mitigation | | — -| — -| — -| — -| | Masked diffusion (F5-TTS) | ✅ Yes | ✅ Exists | Masked speech anchor (partial) | | AdaLN / FiLM (VoxFlash-TTS) | ❌ No (embedding injection) | ❌ None | — | | Perceiver + cross-attention (SupertonicTTS) | ❌ No (independent encoding) | ❌ None | — | | In-context prompt (VALL-E) | ✅ Yes (AR context) | Limited (AR unidirectionality) | Autoregressive single direction |

AdaLN/FiLM and Perceiver + cross-attention are immune by design — the reference is processed independently and never shares a self-attention sequence with the noisy generation region.

VALL-E’s autoregressive nature means reference tokens can only be attended to by subsequent tokens, not the other way around — noise contamination is structurally bounded.

Summary

Reference audio noise contamination in masked diffusion TTS is a structural consequence of global bidirectional self-attention:

• Cause: Reference frames attend to the noisy generation region; noise bleeds into reference representations
• Propagation: Contamination accumulates layer by layer; deep layers see heavily mixed reference representations
• Timing: Worst at early diffusion timesteps, precisely when structural generation decisions matter most
• Mitigation: F5-TTS’s masked speech anchor provides a persistent clean signal, partially counteracting contamination
• Practical impact: Real but not fatal — large-scale training and the anchor mechanism keep the approach effective

If timbre fidelity is the top priority, decoupled approaches (AdaLN/FiLM, Perceiver + cross-attention) eliminate this problem at the architectural level. VoxFlash-TTS is a concrete example: it injects a speaker embedding via AdaLN, the reference audio is encoded in an independent path and never participates in the noisy sequence’s self-attention — reference noise contamination is eliminated by design. If simplicity is the priority, masked diffusion with enough training scale is a viable engineering choice.

References

• F5-TTS: Chen et al. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv:2410.06885
• E2-TTS: Eskimez et al. (2024). E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS. arXiv:2406.18009
• VoiceBox: Le et al. (2023). Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale. NeurIPS 2023
• SupertonicTTS: Kim et al. (2025). SupertonicTTS. arXiv:2503.23108
• VoxFlash-TTS Demo: voxflash.github.io