Why Your Flow Matching TTS Model Won’t Converge — and the Latent Statistics Behind It

A statistical deep dive into VAE latent scale, SNR mismatch, and per-channel normalization, with VoxFlash-TTS as a running case study

If you’re training a Flow Matching speech model and convergence feels slower or shakier than it should, the culprit may not be your architecture at all — it could be the scale of your VAE latents.

This piece walks through the math behind why that happens. Starting from the OT-CFM interpolation path, I derive the mean and variance of both the training input and the target velocity field, show how a weakly-regularized VAE can quietly produce a latent space wildly mismatched in scale with Gaussian noise, and borrow the signal-to-noise-ratio (SNR) framework from image diffusion research to explain exactly how that mismatch hurts optimization. I close with concrete, low-cost engineering fixes — chief among them, per-channel normalization — using VoxFlash-TTS (a 9 Hz ultra-compressed latent TTS system) as a running example throughout.

A caveat up front: several claims here, particularly about VoxFlash-TTS’s actual latent statistics, are theoretical predictions rather than measured facts. I’ll flag these as they come up, and they’re a natural next step for experimental follow-up.

1. Why Does This Problem Arise?

In Flow Matching-based speech synthesis models such as F5-TTS and VoxFlash-TTS, the training input is an interpolation between a VAE latent and Gaussian noise:

xt=t⋅x0+(1−t)⋅ϵ,t∈[0,1]x_t = t \cdot x_0 + (1 — t) \cdot \epsilon, \quad t \in [0, 1]xt​=t⋅x0​+(1−t)⋅ϵ,t∈[0,1]

A natural question follows: what does this interpolation actually look like, statistically — and is that good or bad for training?

If x0x_0 x0​’s distribution differs drastically in scale from ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I) ϵ∼N(0,I), the whole training process unfolds in a severely scale-imbalanced space. That tends to show up as unstable gradients, sluggish convergence, and a model that needs more capacity than it should to compensate.

The root cause traces back to how the VAE itself was trained. Reconstruction-focused VAEs in speech synthesis typically use vanishingly small KL divergence weights, leaving the latent space almost entirely unconstrained — which can produce a latent point cloud whose scale has little to do with that of Gaussian noise.

The rest of this piece works through the statistics step by step and arrives at a small set of engineering fixes.

2. Deriving the Flow Matching Input Distribution

2.1 Mean

E[xt]=t⋅E[x0]+(1−t)⋅E[ϵ]=t⋅E[x0]\mathbb{E}[x_t] = t \cdot \mathbb{E}[x_0] + (1-t) \cdot \mathbb{E}[\epsilon] = t \cdot \mathbb{E}[x_0]E[xt​]=t⋅E[x0​]+(1−t)⋅E[ϵ]=t⋅E[x0​]

If the VAE’s KL regularization is strong enough that the aggregate posterior satisfies E[x0]≈0\mathbb{E}[x_0] \approx 0 E[x0​]≈0, then E[xt]≈0\mathbb{E}[x_t] \approx 0 E[xt​]≈0 too. Under weak regularization, E[x0]\mathbb{E}[x_0] E[x0​] is whatever the data distribution happens to produce, and there’s no reason it should sit near zero.

2.2 Variance

Since x0x_0 x0​ and ϵ\epsilon ϵ are independent:

Var(xt)=t2⋅Var(x0)+(1−t)2⋅Var(ϵ)=t2σx02+(1−t)2\text{Var}(x_t) = t² \cdot \text{Var}(x0) + (1-t)² \cdot \text{Var}(\epsilon) = t² \sigma{x_0}² + (1-t)²Var(xt​)=t2⋅Var(x0​)+(1−t)2⋅Var(ϵ)=t2σx0​2​+(1−t)2

with standard deviation:

σt=t2σx02+(1−t)2\sigmat = \sqrt{t² \sigma{x_0}² + (1-t)²}σt​=t2σx0​2​+(1−t)2​

2.3 Everything Hinges on σx0\sigma_{x_0} σx0​​

This single equation tells you almost everything: the statistical shape of the interpolation is entirely controlled by σx0\sigma_{x_0} σx0​​, the standard deviation of the VAE latent.

tt tσt\sigmat σt​ (σx0=1\sigma{x_0}=1 σx0​​=1)σt\sigmat σt​ (σx0=10\sigma{x_0}=10 σx0​​=10)0.01.001.000.20.832.060.50.715.100.80.838.061.01.0010.00

When σx0≈1\sigma_{x_0} \approx 1 σx0​​≈1, σt\sigmat σt​ stays in [0.71,1.00][0.71, 1.00] [0.71,1.00] across the whole trajectory — bounded, tight, and consistent regardless of tt t. When σx0≫1\sigma{x_0} \gg 1 σx0​​≫1, variance grows nearly linearly with tt t, and the scale at t=0t=0 t=0 versus t=1t=1 t=1 differs by an order of magnitude. That’s a training distribution badly out of balance.

Which raises the obvious next question: what actually determines σx0\sigma_{x_0} σx0​​?

3. Why Weak KL Regularization Causes Latent Scale Imbalance

3.1 Two Variances That Get Confused

Tracking down σx0\sigma_{x_0} σx0​​ means carefully separating two different things that both get called “variance.”

The VAE training objective is:

L=Lrecon+λ⋅DKL(q(z∣x)∥N(0,I))\mathcal{L} = \mathcal{L}{\text{recon}} + \lambda \cdot D{KL}(q(z|x) | \mathcal{N}(0, I))L=Lrecon​+λ⋅DKL​(q(z∣x)∥N(0,I))

For each sample xx x, the encoder outputs a posterior:

q(z∣x)=N(μ(x), σ2(x)⋅I)q(z|x) = \mathcal{N}(\mu(x),\ \sigma²(x) \cdot I)q(z∣x)=N(μ(x), σ2(x)⋅I)

Here’s where the confusion starts:

Posterior variance σ2(x)\sigma²(x) σ2(x) (per-sample) is how uncertain the encoder is about this one sample — the size of the probability “bubble” around it. KL regularization presses directly on this term. As λ→0\lambda \to 0 λ→0, the bubble shrinks toward a single deterministic point μ(x)\mu(x) μ(x).

Dataset variance σx02\sigma_{x_0}² σx0​2​ (population-level) is how spread out all the bubbles’ centers are across the dataset — {μ(x1),μ(x2),…,μ(xN)}{\mu(x_1), \mu(x_2), \ldots, \mu(x_N)} {μ(x1​),μ(x2​),…,μ(xN​)} forming a point cloud in latent space. This is shaped by the diversity of the data itself, and the KL weight barely touches it.

ConceptWhat it measuresUnder weak KLPosterior variance σ2(x)\sigma²(x) σ2(x)Per-sample uncertainty≈0\approx 0 ≈0 — nearly deterministicDataset variance σx02\sigma_{x_0}² σx0​2​Point cloud spreadSet by data diversity, often large

The short version: weak KL means each point is sharp, but the cloud as a whole can sprawl. Strong KL means each point is a bit blurry, but the cloud stays compressed near N(0,I)\mathcal{N}(0,I) N(0,I).

3.2 What Happens at Vanishingly Small KL Weights

Reconstruction-focused VAEs in speech synthesis commonly use λ=10−5\lambda = 10^{-5} λ=10−5 to 10−610^{-6} 10−6 to squeeze out maximum reconstruction fidelity.

In theory, this pushes three things in predictable directions:

• Posterior variance σ2(x)→0\sigma²(x) \to 0 σ2(x)→0: sampling z=μ(x)+σ(x)ϵ≈μ(x)z = \mu(x) + \sigma(x)\epsilon \approx \mu(x) z=μ(x)+σ(x)ϵ≈μ(x), so the VAE behaves almost like a deterministic autoencoder
• Encoded mean μ(x)\mu(x) μ(x): free to go wherever reconstruction loss pushes it, faithfully tracking real data structure
• Point cloud variance σx02\sigma_{x_0}² σx0​2​: with the encoder unconstrained, different samples can land anywhere across a wide, irregular region of latent space

VoxFlash-TTS combines an extremely high compression ratio (9 Hz) with weak KL regularization — exactly the conditions that, in theory, tend to inflate aggregate posterior variance. Whether σx0\sigma_{x_0} σx0​​ is actually much larger than 1 in practice is an empirical question that requires measuring the full training set; everything from here on assumes this theoretical picture holds, pending that measurement.

4. The Velocity Field Target: Statistics of x0−ϵx_0 — \epsilon x0​−ϵ

4.1 Mean and Variance

Under OT-CFM, the model is trained to predict the velocity field:

vθ(xt,t)≈v=x0−ϵv_\theta(x_t, t) \approx v = x_0 — \epsilonvθ​(xt​,t)≈v=x0​−ϵ

By independence of x0x_0 x0​ and ϵ\epsilon ϵ:

E[v]=E[x0]\mathbb{E}[v] = \mathbb{E}[x_0]E[v]=E[x0​] Var(v)=Var(x0)+Var(ϵ)=σx02+1\text{Var}(v) = \text{Var}(x0) + \text{Var}(\epsilon) = \sigma{x_0}² + 1Var(v)=Var(x0​)+Var(ϵ)=σx0​2​+1

The velocity field’s variance is always larger than the latent’s own variance — by exactly 1, the noise’s contribution.

4.2 Two Scenarios, Worlds Apart

σx0≈1\sigma_{x_0} \approx 1 σx0​​≈1 — latent and noise on matched scales

v∼N(0, 2),σv=2≈1.41v \sim \mathcal{N}(0,\ 2), \quad \sigma_v = \sqrt{2} \approx 1.41v∼N(0, 2),σv​=2​≈1.41

A compact, bounded regression target — exactly what neural networks are good at fitting.

σx0≫1\sigma_{x0} \gg 1 σx0​​≫1 — say, σx0=10\sigma{x_0} = 10 σx0​​=10

σv=101≈10.05\sigma_v = \sqrt{101} \approx 10.05σv​=101​≈10.05

Now Var(x0)=100\text{Var}(x_0) = 100 Var(x0​)=100 swamps Var(ϵ)=1\text{Var}(\epsilon) = 1 Var(ϵ)=1. The training target is dominated by the latent itself; noise contributes roughly 1% of the total variance and is essentially along for the ride. The model has to predict large-magnitude vectors, the loss operates at a much larger absolute scale, and convergence suffers.

4.3 One Number Governs Both Ends

Notice that input variance and output variance are governed by the exact same quantity:

σxt2=t2σx02+(1−t)2,σv2=σx02+1\sigma_{xt}² = t² \sigma{x_0}² + (1-t)², \quad \sigmav² = \sigma{x_0}² + 1σxt​2​=t2σx0​2​+(1−t)2,σv2​=σx0​2​+1

This is the whole engineering payoff in one observation: normalizing the latent fixes the input distribution and the output distribution at the same time, with a single operation.

5. What Makes a Distribution Training-Friendly?

Putting the pieces together, when σx0≈1\sigma_{x_0} \approx 1 σx0​​≈1:

• Input: σxt∈[0.71,1.00]\sigma_{x_t} \in [0.71, 1.00] σxt​​∈[0.71,1.00] across the board — bounded, consistent regardless of tt t, gradients more stable
• Output: σv=2\sigma_v = \sqrt{2} σv​=2​, fixed and well-behaved — an easier regression target
• SNR curve: SNR(t)=t2/(1−t)2\text{SNR}(t) = t²/(1-t)² SNR(t)=t2/(1−t)2, monotonic and clean

A more stable input, a more regular output, and steadier gradients all fall out of the same single condition: σx0≈1\sigma_{x_0} \approx 1 σx0​​≈1.

One caveat worth stating plainly: σx0≈1\sigma_{x_0} \approx 1 σx0​​≈1 hasn’t been proven globally optimal in any rigorous sense. It’s a statistically reasonable, empirically training-friendly state — one where the spread of the latent matches the scale of standard Gaussian noise, sidestepping the imbalance that otherwise creeps into training.

6. SNR Mismatch: Why Scale Imbalance Actually Hurts

So far we’ve shown that as σx0\sigma_{x_0} σx0​​ grows, both the input distribution and the velocity field target inflate in scale. But what does that do to optimization, concretely? Image generation research has a well-developed answer: the signal-to-noise ratio (SNR) framework.

6.1 Defining SNR

In the diffusion / Flow Matching setting:

SNR(t)=αt2⋅σx02σt2\text{SNR}(t) = \frac{\alphat² \cdot \sigma{x_0}²}{\sigma_t²}SNR(t)=σt2​αt2​⋅σx0​2​​

With αt=t\alpha_t = t αt​=t under OT-CFM’s linear interpolation, this becomes:

SNR(t)=t2σx02(1−t)2\text{SNR}(t) = \frac{t² \sigma_{x_0}²}{(1-t)²}SNR(t)=(1−t)2t2σx0​2​​

SNR captures how much the signal (the latent) dominates over noise at a given timestep tt t.

6.2 The Lesson from Image Generation

Simple Diffusion (Hoogeboom et al., 2023) and related work showed that high-resolution image generation is harder, and traced part of the cause to a shift in SNR structure. As resolution increases — larger latent dimension DD D — uneven channel variances cause the SNR curve to drift differently across channels: high-variance channels run “hot” (elevated SNR) at a given tt t, low-variance channels run “cold.” The model ends up facing a different noise environment in every channel.

Image domainSpeech analogHigher resolution → larger latent dim DD DLonger utterance → larger time dim TT TUneven variance across spatial positionsUneven variance across latent channelsSNR structure shifts with resolutionSNR structure shifts with sequence length / channel count

The framework transfers almost without modification — “resolution” becomes “sequence length × compression ratio.”

6.3 What This Costs You at the Optimization Level

Gradient scale imbalance. Under a shared MSE loss, high-SNR channels (signal-dominated) produce much larger gradients than low-SNR channels (noise-dominated). The high-variance channels end up steering the parameter updates; the low-variance ones get drowned out.

Adam can only do so much. Adaptive optimizers (Kingma & Ba, 2015) compensate via second-moment estimates, but they need many steps to learn the true per-channel gradient scale — early training is correspondingly inefficient.

Training stability suffers. When SNR varies too wildly across tt t, the model struggles to find a single parameter configuration that works for both the high-noise and low-noise regimes, and convergence drags.

7. Where Speech Latents Run Into SNR Trouble: Channels and Compression

With the general SNR mismatch story in hand, let’s get concrete about speech latents specifically.

7.1 Uneven Channel Variance

In a weakly-regularized VAE, the encoder is free to organize information however it likes. Different latent channels often end up encoding different semantic layers — fundamental frequency, formant envelope, vocal tract shape — and their variances can diverge substantially:

σch12≠σch22≠⋯≠σchC2\sigma²_{\text{ch}1} \neq \sigma²{\text{ch}2} \neq \cdots \neq \sigma²{\text{ch}_C}σch1​2​=σch2​2​=⋯=σchC​2​

If channel c1c_1 c1​ has σ12=100\sigma_1² = 100 σ12​=100 and channel c2c_2 c2​ has σ22=1\sigma_2² = 1 σ22​=1, their velocity field variances are:

Var(v(c1))=101,Var(v(c2))=2\text{Var}(v^{(c_1)}) = 101, \quad \text{Var}(v^{(c_2)}) = 2Var(v(c1​))=101,Var(v(c2​))=2

Two completely different SNR curves, one shared loss function. The model has no choice but to compromise between two channels that effectively live in different noise regimes.

This isn’t unique to deep latents, either. Mel-spectrogram-based systems like F5-TTS see something similar: low-frequency bands carry concentrated energy, high-frequency bands are sparse, and that asymmetry shows up as a systematic variance difference across frequency channels.

7.2 Compression Ratio and Velocity Field Smoothness

Flow Matching implicitly assumes a smooth velocity field — the underlying ODE needs to be well-behaved. At high compression ratios, each latent frame packs in more semantic content, so neighboring frames jump around more, and the velocity field becomes more jagged along the time axis.

VoxFlash-TTS runs at 9 Hz — far more compressed than mel-based systems like F5-TTS (~80 Hz). Practically, that means:

• The Euler solver, at a fixed NFE (16 in VoxFlash-TTS’s case), accumulates larger discretization error
• More model capacity is needed to track a genuinely more complex velocity field
• Long utterances may show worse SNR mismatch than short ones

ScenarioSNR mismatch riskUniform channel variance + low compressionLowest — close to the friendly regimeUniform channel variance + high compressionComplex velocity field, but channels stay SNR-alignedUneven channel variance + low compressionCross-channel mismatch, but smooth in timeUneven channel variance + high compressionWorst case — both problems compound

7.3 Sequence Length

When training data mixes long and short utterances, latent sequence length TT T — and the total dimension D=C×TD = C \times T D=C×T — varies sample to sample. A model trained mostly on short clips, then pointed at a long utterance at inference time, may run into an SNR structure it hasn’t really seen. The resulting quality drop isn’t purely a capacity problem; part of it is this kind of statistical mismatch.

8. The Fix: Why Normalization Works

8.1 Per-Channel Normalization, Mechanically

This is the cheapest, most direct fix available. Before Flow Matching training begins, compute per-channel statistics over the training set latents and rescale linearly:

μc=Ex∈D[x0(c)],σc=E[x0(c)2]−μc2\muc = \mathbb{E}{x \in \mathcal{D}}[x_0^{©}], \quad \sigma_c = \sqrt{\mathbb{E}[{x_0^{©}}²] — \mu_c²}μc​=Ex∈D​[x0(c)​],σc​=E[x0(c)​2]−μc2​​ x~0(c)=x0(c)−μcσc,c=1,…,C\tilde{x}_0^{©} = \frac{x_0^{©} — \mu_c}{\sigma_c}, \quad c = 1, \ldots, Cx~0(c)​=σc​x0(c)​−μc​​,c=1,…,C

Every channel now has unit variance, and every SNR curve aligns:

SNR(c)(t)=t2⋅1(1−t)2=t2(1−t)2,∀c\text{SNR}^{©}(t) = \frac{t² \cdot 1}{(1-t)²} = \frac{t²}{(1-t)²}, \quad \forall cSNR(c)(t)=(1−t)2t2⋅1​=(1−t)2t2​,∀c

One distinction matters a lot here: these are global, dataset-level statistics — not per-sample ones. Global normalization is just a linear change of coordinates; it preserves the relative structure between samples entirely, so the decoder can still tell them apart. Normalizing per sample, by contrast, would wipe out exactly the information that makes reconstruction possible.

Stable Diffusion’s scaling factor of 0.18215 (Rombach et al., 2022) is, in essence, 1/σglobal1/\sigma_{\text{global}} 1/σglobal​ — their VAE latent mean happens to sit near zero, so only the standard deviation division is needed.

The full inference pipeline:

ϵ∼N(0,I)→Flow Matchingx~0→×σc+μcx0→VAE Decodermel/audio\epsilon \sim \mathcal{N}(0, I) \xrightarrow{\text{Flow Matching}} \tilde{x}_0 \xrightarrow{\times \sigma_c + \mu_c} x_0 \xrightarrow{\text{VAE Decoder}} \text{mel/audio}ϵ∼N(0,I)Flow Matching​x~0​×σc​+μc​​x0​VAE Decoder​mel/audio

8.2 What This Buys You in Model Capacity

Without per-channel normalization, the model is implicitly learning two things at once:

vθ(xt,t)=fscale(c)⏟channel scale compensation — easy+fcontent(xt,t)⏟speech content modeling — hardv_\theta(xt, t) = \underbrace{f{\text{scale}}©}{\text{channel scale compensation — easy}} + \underbrace{f{\text{content}}(xt, t)}{\text{speech content modeling — hard}}vθ​(xt​,t)=channel scale compensation — easyfscale​©​​+speech content modeling — hardfcontent​(xt​,t)​​

Normalization strips fscalef{\text{scale}} fscale​ out entirely, turning it into a fixed preprocessing step, and frees the model to spend its full capacity on fcontentf{\text{content}} fcontent​.

In practice, that means a model trained on un-normalized latents needs more capacity, more training time, or more data to reach the same quality bar as one trained on normalized latents. Per-channel normalization costs zero parameters and buys a meaningful improvement in how efficiently those parameters get used.

8.3 A Few Complementary Levers

Raise the KL weight. Pushing λ\lambda λ up (say, from 10−610^{-6} 10−6 to 10−410^{-4} 10−4) lets KL regularization actively pull the point cloud inward, tackling σx0≫1\sigma_{x_0} \gg 1 σx0​​≫1 at its source. The cost is reconstruction quality, so this needs careful tuning.

Min-SNR loss weighting (Hang et al., 2023): reweight the loss across tt t:

L=Et[min⁡(SNR(t), γ)⋅∥vθ(xt,t)−v∥22]\mathcal{L} = \mathbb{E}t \left[ \min(\text{SNR}(t),\ \gamma) \cdot |v\theta(x_t, t) — v|_2² \right]L=Et​[min(SNR(t), γ)⋅∥vθ​(xt​,t)−v∥22​]

This caps the weight given to high-SNR regions, so the model doesn’t get to coast through the easy, low-noise part of training while neglecting the hard, high-noise part — especially useful for long utterances and high compression ratios.

Input-side normalization (a supplement, not a substitute). Normalizing raw training data before it hits the VAE stabilizes training and speeds convergence, but the encoder is nonlinear, so this only indirectly shapes the latent distribution. It pairs well with latent-level normalization but doesn’t replace it.

8.4 Where to Spend Your Time First

Priority 1: Per-channel latent normalization  ← zero parameter cost, most reliable, do this first
Priority 2: Input-side global normalization    ← cheap stability win
Priority 3: Raise the KL weight                ← task-dependent trade-off, tune carefully
Priority 4: Min-SNR loss weighting             ← worth it for long utterances / high compression

9. Case Study: Applying This to VoxFlash-TTS

VoxFlash-TTS’s architecture happens to check most of the boxes that make this problem worse in theory:

• 9 Hz ultra-compressed latent — far more compressed than mel-based systems like F5-TTS (~80 Hz), with high per-frame signal power putting real strain on the velocity field’s smoothness assumption
• Weak KL regularization — the VAE is optimized for reconstruction first, leaving the latent point cloud largely unconstrained; the actual magnitude of σx0(c)\sigma_{x_0}^{©} σx0​©​ and how much it varies across channels is an open empirical question
• Mixed-length training data — different sequence lengths imply different SNR structures, creating real risk of mismatch on long-form inference

A practical path forward:

1. Measure first. Compute per-channel mean and standard deviation across the training set latents. This is step zero — it tells you how serious the problem actually is before you do anything else.
2. Normalize if warranted. If channel variances differ substantially, apply per-channel normalization to bring each channel to mean ≈0\approx 0 ≈0, variance ≈1\approx 1 ≈1.
3. Bake the parameters in. Save {μc,σc}c=1C{\mu_c, \sigmac}{c=1}^C {μc​,σc​}c=1C​ as fixed constants and wire inverse normalization into the ONNX inference pipeline, right before the VAE decoder.
4. Test on long utterances specifically. Bucket evaluation samples by length. If long-form generation noticeably underperforms, Min-SNR-γ\gamma γ weighting is a reasonable next experiment.
5. Let Sway Sampling do its job. Once channels are SNR-aligned, Sway Sampling’s bias toward mid-range tt t values targets the genuinely hardest part of the trajectory — exactly as intended.

Worth saying plainly: none of this has been validated experimentally yet. Latent variance statistics, loss curves with and without normalization, and long-vs-short utterance quality comparisons are the obvious next steps.

10. Wrapping Up

The chain of reasoning here looks like this:

λKL≪1⏟weak VAE regularization⇒σ2(x)→0⏟near-deterministic encoding+σx02 potentially≫1⏟scattered point cloud (unverified)⇒cross-channel SNR mismatch⏟gradient imbalance⇒slow, unstable convergence⏟\underbrace{\lambda{\text{KL}} \ll 1}{\text{weak VAE regularization}} \Rightarrow \underbrace{\sigma²(x) \to 0}{\text{near-deterministic encoding}} + \underbrace{\sigma{x0}² \text{ potentially} \gg 1}{\text{scattered point cloud (unverified)}} \Rightarrow \underbrace{\text{cross-channel SNR mismatch}}{\text{gradient imbalance}} \Rightarrow \underbrace{\text{slow, unstable convergence}}{ }weak VAE regularizationλKL​≪1​​⇒near-deterministic encodingσ2(x)→0​​+scattered point cloud (unverified)σx0​2​ potentially≫1​​⇒gradient imbalancecross-channel SNR mismatch​​⇒slow, unstable convergence​​

And the fix:

per-channel normalization⏟free⇒σx0(c)≈1, ∀c⏟SNR alignment⇒σxt∈[0.71,1], σv=2⏟tight, regular distributions⇒balanced gradients, faster convergence⏟\underbrace{\text{per-channel normalization}}{\text{free}} \Rightarrow \underbrace{\sigma{x0}^{©} \approx 1,\ \forall c}{\text{SNR alignment}} \Rightarrow \underbrace{\sigma_{x_t} \in [0.71, 1],\ \sigmav = \sqrt{2}}{\text{tight, regular distributions}} \Rightarrow \underbrace{\text{balanced gradients, faster convergence}}_{ }freeper-channel normalization​​⇒SNR alignmentσx0​©​≈1, ∀c​​⇒tight, regular distributionsσxt​​∈[0.71,1], σv​=2​​​⇒balanced gradients, faster convergence​​

Zooming out: Flow Matching doesn’t strictly require latents to follow a standard Gaussian distribution. What it actually wants is for the training data and the noise to live at comparable statistical scales. Per-channel normalization doesn’t touch the semantics encoded in the latent at all — it just redefines the coordinate system, freeing the model to spend its limited capacity learning the real data distribution instead of compensating for arbitrary scale differences between channels. Stable Diffusion already builds this idea into its scaling factor. It deserves the same attention in high-compression speech latent modeling.

References

1. Lipman, Y., et al. “Flow Matching for Generative Modeling.” ICLR 2023.
2. Albergo, M. S., & Vanden-Eijnden, E. “Building Normalizing Flows with Stochastic Interpolants.” ICLR 2023.
3. Hoogeboom, E., et al. “Simple Diffusion: End-to-End Diffusion for High Resolution Images.” ICML 2023.
4. Hang, T., et al. “Efficient Diffusion Training via Min-SNR Weighting Strategy.” ICCV 2023.
5. Rombach, R., et al. “High-Resolution Image Synthesis with Latent Diffusion Models.” CVPR 2022.
6. Kingma, D., & Ba, J. “Adam: A Method for Stochastic Optimization.” ICLR 2015.
7. Kingma, D., & Welling, M. “Auto-Encoding Variational Bayes.” ICLR 2014.
8. Chen, X., et al. “F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching.” arXiv 2410.06885.
9. VoxFlash-TTS, GitHub: github.com/VoxFlash/VoxFlashTTS

For the full version of this article with natively-rendered LaTeX equations, see the companion post on Hashnode: Why Your Flow Matching TTS Model Won't Converge — and the Latent Statistics Behind It