Diffusion’s Hidden Dimension: Why “What We Generate First” Changes Everything

In a recent paper from Fei-Fei Li’s team, the core conversation shifts toward a fundamental but often overlooked question in diffusion models: Does the order of “what gets generated first” dictate the final quality?

In the standard Latent Diffusion (LDM) pipeline, we typically compress an image into a latent space using an encoder, perform the diffusion process there, and then reconstruct the pixels via a pre-trained decoder. While computationally efficient, this comes with a clear trade-off. The encoder inevitably loses information, and the model ends up learning the distribution of the latent space rather than the raw pixels. High compression makes diffusion easier to learn but lowers the performance ceiling; low compression raises the ceiling but makes optimization a nightmare.

Li’s team proposes a way to have it both worlds — retaining the end-to-end modeling power of pixel space while capturing the fast convergence of latent methods. Their solution isn’t to change the modeling space, but to change the modeling sequence.

Decoupling the Generation Flow

The architecture utilizes a single DiT (Diffusion Transformer) to handle two modalities simultaneously: raw pixel patches and latent representations extracted from a self-supervised DINOv2 model.

Technically, the pixel and latent embeddings are simply summed together. However, the researchers maintain distinct output heads to ensure the two flows remain decoupled. The brilliance lies in the training: each modality has its own independent time variable. They are noisified and denoised separately, with the final loss being a weighted sum of both. During inference, the latent branch acts as a “scaffolding” to help the model organize structure first, while only the pixel results are kept for the final output.

The Power of SNR and Scaling

One of the most insightful takeaways from the paper is the use of Signal-to-Noise Ratio (SNR) to characterize the sequence of information emergence. A higher SNR indicates a clearer modality. Whichever modality sees its SNR rise faster is, in effect, being “generated first.”

Interestingly, the team proved that adjusting the numerical scale of a modality (multiplying raw data by a scalar) is mathematically equivalent to shifting its time schedule. If you increase the scale, that modality achieves a higher SNR at the same timestep $t$. This means it starts denoising earlier and carries more “useful” information in the initial stages of generation. This suggests that many empirical “tricks” used in existing latent models are actually just implicit ways of reordering the denoising sequence.

Finding the Optimal Trajectory

Through experiments testing various generation trajectories, the team found a consistent pattern: as long as the latent space begins denoising before the pixel space, FID scores drop significantly. Conversely, if pixels are forced to become clear first, performance collapses.

The intuition is straightforward: the model needs to settle on the semantic “skeleton” via latents before it starts filling in the “flesh” of pixel-level details.

[Image comparing different diffusion trajectories: latent-first vs pixel-first]

Under a fixed schedule, a cascaded sequence yielded the best results. In this setup:

1. Latent Phase: Pixel noise is held at $t=0$ while the latent $t$ transitions from $0$ to $1$.
2. Pixel Phase: The latent is held at $t=1$ (fully formed) while the pixel $t$ transitions from $0$ to $1$.

Whether in conditional or unconditional generation, the gains from this specific ordering far outweighed the benefits of simple distillation or additional supervision. It suggests that “sequence” isn’t just a hyperparameter — it is a foundational mechanism for how generative models learn to see.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)