Why Deepmind’s Unified Latents Are The Beautiful New Breakthrough in Gen AI

TLDR: *The extremely succesful and SOTA Stable Diffusion GenAI architecture uses a decoder (usually a diffusion model) on top of a frozen encoder (could be just a CNN). While this works very well, all version of this have one mostly overlooked inherent flaw: The frozen encoder doesn’t know how the decoder (diffusion) works, which is why it might compress information in a way that is “hard” or unintuitive for the Diffusion model to understand and denoise (because it is frozen, meaning it was aready trained and is just used for inference now). Unified Latents moves us away from this chaining of different models and toward a single, cohesive, mathematical (training) objective.*

First let’s start with the basics.

Stable Diffusion models were (in 2022) the breakthrough behind text-to-image generation and image/video generation, like DALLE. It is basically a first trained, then frozen Variational Auto Encoder (VAE), which is a model that has an encoder which compresses the input image into a latent space from which then the decoder tries to reconstruct the input image.

The basic scheme of a variational autoencoder. The model receives as input. The encoder compresses it into the latent space. The decoder receives as input the information sampled from the latent space and produces as similar as possible to. By EugenioTL — Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=107231101

Variational refers to the additional condition that the distribution of the latent space must always follow a certain (noisy gaussian) distribution. This is a really neat mathematical trick to make sure that our latents are mapped into a clean, well clustered and defined, continuos, space. This then makes it (a) easier for the decoder and (b) allows us to sample out of the distribution of latents, which means we can create synthetic data similar to the training data. Specifically, this means that instead of having the encoder output a simple latent vector z, we let it output a vector of the means and a vector of standard deviations, from which we can sample. This way, the decoder is forced to reconstruct/learn the whole continuous neighbouhood of the input data.

If we use a Kullback–Leibler (KL) divergence loss to achieve this, we can even guarantee an “evidence” lower bound (see ELBO) on the probability of our data. KL divergence (also called relative entropy and I-divergence) is a statistical distance, or similarity measure of two probability distributions.

After the VAE encoder produces the compressed latent space the diffusion process is added. First, a gaussian noise is iteratively applied to the compressed latent representation during **forward diffusion. Then, via multiple U-Net blocks which are composed of a ResNet backbone and cross-attention blocks helping the model to gain a relational understanding of the image via conditioning, the noisy latent is denoised to obtain a “clean” latent again. But what is conditioning? It is a way to help and guide the denoising steps by essentially adding extra information, e.g. textual informatio via CLIP embeddings or other images, but all translated via some encoder(s) into the same latent dimensions. The information is then added via cross-attention blocks (remeber “Attention Is All You Need”? ;)). In 2026 the industry has moved away from the traditional U-Net backbone toward [Diffusion Transformers (DiT).](https://arxiv.org/abs/2212.09748)**

The Stable Diffusion architecture from 2022. From wikipedia.

Finally, the VAE decoder generates the final image by converting the representation back into pixel space. In Stable Diffusion, the VAE decoder is usually a standard CNN. In UL, the Decoder is itself a Diffusion model.

Important note on the actual training process:

The VAE is traditionally trained first and then only used with frozen weights, so only doing inference and not retraining. Only after it is frozen is the separate diffusion U-NET model trained, working basically only on the trained VAE’s specific latent space. This “one-size-fits-all” latent space isn’t necessarily optimized for the specific diffusion model that will follow. This standard VAE archtiecture then effectively leads to a tradeoff: either the latent space is a mess (bad for sampling), or the reconstructions are blurry (bad for quality).

.. This leaves room for optimization, right …?

The intial approach to solve this tradeoff was manual tuning. Researchers introduced the hyperparameter beta. We add a weight beta to the KL divergence term in the loss function, and essentially “punish” the encoder if it tries to make the latent space too complex. This forces the model to find the most efficient, independent variables of the data and map them to separate dimensions in the latent space.

However, this created a massive trade-off. A high beta value gives you a beautifully organized, “disentangled” latent space where every dimension means something, but it often results in blurry, “soupy” reconstructions because the encoder is too scared to pass through fine details.

This is exactly the bottleneck UL solves!

Now we can start to have a look at how Google’s Deepmind team in Amsterdam, which identified this bottleneck, propose to improve this.

UL proposes that in order to get better latents they are “jointly regularized by a diffusion prior and decoded by a diffusion model. By linking the encoder’s output noise to the prior’s minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate” (Deepmind Amsterdam https://arxiv.org/pdf/2602.17270, Feb 26). Check out their paper!

In the UL paper (Feb 2026), they realized that the VAE noise (from step 1) and the Diffusion noise (from step 2) are mathematically the same thing: Gaussian blur. Instead of having two separate noise processes, UL says: “Why have the VAE encoder output a mean and std dev (so distribution — which is hard to train) when we can just have the encoder output a clean point, and let the Diffusion Prior define the noise level?”

So the main idea of Unfied Latents UL ist that instead of doing two disconnected training stages of first VAE and then Diffusion, we train only once in a unified way. Reminder: This allows us to get a latent space (output of endoder) which is optimized for the diffusion process on the latents before decoding again. So the Encoder is trained jointly with the generative model, and the Diffusion Prior (DP) replaces the simple KL-divergence loss. Rather than forcing latents into a simple Gaussian “blob,” we let a Diffusion model learn the complex, actual “shape” of the latents. Therfore we have a better latent space for the decoder to work with.

gaussian distribution like in VAEs. The framework then adds a fixed amount of Gaussian noise to it (linked to the prior’s minimum noise level) to get a noisy latent vector z_t. Then, we replace the standard Stable Diffusion VAE decoder (a CNN) with a Diffusion model. Btw, moving from a CNN to a Diffusion encoder means moving from a simple reconstruction, decompressor model to a conditional generative model. The “noisy latent” vector z_t is then what the DP learns to model and the Decoder learns to reconstruct ( so we train everything at the same time).

The DP learns a high-level, semantic representation vector of the image containing the “essence”. The DP provides the global structure, and the Decoder uses its own generative power to Hallucinate the Fine Details that aren’t explicitly in the global structure. The Diffusion Prior (DP) essentially acts as the Generative Latent Prior. In standard SD, the latent space is often assumed to be normally distributed. In UL, the DP learns the actual distribution of encoded images, making the “shortcut” between encoding and decoding much more semantically aware.

Since this is IMO the crucial part of the paper, I’ll try to explain it further from different point-of-views:

Loss POV: This is the “unified part” of UL. The DP and the Decoder are connected via both their combined loss functions. This means the Encoder is constantly receiving two sets of instructions from the loss during training. From the DP: “Make the latent more predictable/orderly so I can model it!”. And from the Decoder: “Make the latent more descriptive so I can draw the pixels!”. So we have a cohesive mathematical (training) objective.

Architectural POV: In other terms, the Diffusion Prior is the “Blueprint” for the Decoder, which is the “Builder”. So the Decoder takes the noisy pixels. It runs them through a Transformer or U-Net block. Inside that block, there is a Cross-Attention layer. Queries Q come from the noisy pixel features. Keys K and Values V come from the “clean” high level semantic representation latent from the DP. The pixels “ask” the latent: “I’m at coordinate (x,y) what color should I be?” The latent answers: “You are part of a golden retriever’s ear; be golden and fuzzy.”

The paper expresess and visualizes this process concisely as follows:

Now to connect the dots to the standard Stable Diffusion architecture: In standard Stable Diffusion we do conditioning by inputting embeddings, e.g. From Text Embeddings from a CLIP encoder, into the Cross-Attention blocks of the Denoising U-Net. In the UL Architecture instead, the Decoder’s Cross-Attention blocks look at the high level semantic Latent like described above.

So why is this so beatiful? And why would it even be a breakthrough?

Rather than the encoder learning to be noisy and outputting a distribution (as in a traditional VAE), the UL framework enforces a fixed noise floor that the encoder must respect. This is achieved via a Maximum Precision Link, a mathematical constraint where we inject a specific, constant amount of Gaussian noise into the latent space during training. This means the encoder is effectively “capped.” It knows exactly how much detail it can send before that detail is washed out by the noise. This forces the encoder to be extremely efficient, focusing only on the most important structural features. This setup solves the Rate-Distortion tradeoff: we no longer have to manually tune the KL-weight to balance blurriness versus structure. The noise floor dictates the bitrate naturally.

From a performance standpoint, the UL framework is a beast. Because the Prior and the Decoder are both Diffusion-based and utilize Transformer (DiT) backbones, the entire architecture is highly optimized for modern GPU clusters.

However inference time is theoretically longer, because we now have two diffusion processes (one in latent space (diffusion prior) and one in pixel space (decoder)), as opposed to a one-shot VAE. Also jointly training two diffusion models is complex and requires a lot of VRAM. However the authors explore first training the encoder using a diffusion prior while placing l2 regularization on the decoder and different loss weightings, which slighty decreased performance.

So, what’s next?

The authors of UL mention that the next frontier is Universal Multimodality. Because this framework treats latents as a “unified language,” they envision a single UL model that doesn’t just do text-to-image, but can seamlessly bridge Video, 3D geometry, and Audio into the same latent manifold.

Stay curious!