Deep Generative Models: From Noise to Structure

I have been watching the CS236 Stanford lectures on YouTube, and this post is a structured set of notes on deep generative models, with a deeper dive into diffusion models and score-based generative models.

Discriminative vs Generative Models

At a high level, discriminative and generative models differ fundamentally in terms of what probabilities they choose to model.

Chain Rule

Discriminative models focus on predicting labels given data, modelling the conditional distribution p(y | x).

• They learn the decision boundary between classes.
• This can be thought of as learning the relationships amongst the inputs xi, followed by a final prediction of the label y

Generative models focus on modelling the joint probability p(y, x), which represents how data xand labels y are generated.

• They learn about the underlying distribution of the data.
• This can be thought of as starting from the label yand generating the data xi step by step.

Stanford CS236: Deep Generative Models | 2023 | Lecture 2 — Background

Different Types of Generative Models

At their core, generative models try to learn the data distribution p(x) so that they can generate new samples that look like the training data.

However, modelling the probability distribution directly is difficult because:

1. the probabilities must be non-negative
2. the sum over all the probabilities must sum up to 1
3. integrating over all possible x values is often intractable in high dimensions

Hence, this has led to the development of different kinds of generative models.

Autoregressive Models

Autoregressive models use the chain rule to decompose a complex joint distribution into a sequence of simpler conditional distributions.

Basically, the probability of an element is conditioned on everything that came before it.

This makes likelihood compoutation tractable and exact, but it comes at a cost: sampling is inherently sequential, which can be slow.

Flow-Based Models

Flow-based models learn deterministic and invertible mappings between latent variables z and the data space x, enabling exact likelihood computation via the change-of-variables formula.

However, the constraint that the transformation must be invertible limits architectural flexibility

Variational Autoencoders

VAEs introduce latent variables zand uses an encoder to output a probability distribution (mean and variance) for the latent space, rather than a fixed point.

Sampling from this probability and decoding it allows the model to generate new data.

Energy-Based Models

EBMs assign an “energy” value to each input, where low energy corresponds to high probability regions.

Z(θ) is the sum/integral over every possible configuration of x, which acts as a normalisation constant to ensure that the area under the probability curve is exactly 1. However, calculating Z(θ) is intractable in high dimensions.

Generative Adversarial Networks

Instead of modelling explicit likelihoods, GANs frame learning as a game between two networks: a generator that tries to produce realistic samples and a discriminator that tries to distinguish real from fake data.

Through this adversarial process, the generator learns to produce increasingly realistic outputs. GANs are known for producing sharp images, but they suffer from training instability and issues like mode collapse.

Diffusion Models

This is the core idea of diffusion models: instead of generating data directly, we define a process that gradually destroys data by adding noise, and then we learn to reverse this process.

Instead of directly predicting the final image, the model is trained to predict the noise that was added during the forward process. This reparameterisation simplifies learning significantly, since predicting noise is statistically easier than reconstructing high-dimensional structure directly.

To understand the math in depth, I highly recommend this video: Diffusion Models | Paper Explanation | Math Explained.

Forward Process

In the forward process, we gradually add Gaussian noise to the image xover time.

How diffusion models work: the math from scratch

Each step in the forward process is defined using a normal distribution N , parameterised by a mean and variance. It takes in xt-1 as input and outputs xt , the image at the next time step with noise added to it.

• βt (noise schedule): determines the amount of noise added at step t
• sqrt(1-βt): scales the previous image to prevent the variance from growing infinitely, keeping the data centered as it becomes more noisy

To generate a sample X from a normal distribution with a specific mean μ and standard deviation σ:

where Z is a random sample from the standard normal distribution Z ~ N(0, 1)

By substituting the mean and variance values from q(xt|xt-1), we get:

By multiplicating q(xt|xt-1)at every step from x0 to xT, the entire process is a Markov chain:

Eventually, xT is equivalent to pure noise from a Gaussian distribution.

Reparameterisation Trick

Instead of going through every intermediate step, we can derive an expression that allows us to jump directly from x0 to xT.

By defining:

We can rewrite the forward process as:

After expanding recursively, all noise terms are summed up into a single Gaussian ϵ:

This gives us a closed form equation:

In training, this allows us to sample xt directly without needing to simulate all steps. We can randomly choose timesteps during training and compute corresponding noise prediction targets efficiently.

Backward Process

In the backward process, the model learns to ‘undo’ the noise added during the forward process.

How diffusion models work: the math from scratch

While the forward process is fixed, the backward process is a learned Markov chain that transforms pure Gaussian noise xT back into a clean data sample x0 .

The goal of the network is to predict the mean of the Gaussian distribution that would shift the noisy image xt closer to its original state xt-1. The variance Σ is fixed or learnt.

Variational Lower Bound

We ultimately want to learn a model that assigns high probability to real data:

But in diffusion models, x0 is connected to a whole chain of latent steps x0 ↔ x1 ↔ x2 ↔ … ↔ xT . Direct computation is intractable because we need to integrate over all possible latent steps.

In diffusion models, we define the variational lower bound based on the KL divergence between the forward process q and the reverse process p .

From here, p and q are substituted in:

The t=1 term is separated out in the summation :

They condition the forward process on x0 :

Converting the log ratios into KL divergences:

• LT: this term ensures that the distribution of the fully noised data after T steps really looks like pure Gaussian noise
• Lt-1 : this term ensures that the model correctly reverses one step of the forward process
• Lo : this term corresponds to reconstructing the original data from the first noisy

In Lt-1 , both terms q(xt-1 | xt, x0) and pθ(xt-1 | xt) are Gaussian.

Using the reparameterisation trick earlier,

In the end, the variational lower bound reduces to a mean squared error between the noise added in the forward process and the noise predicted by the model.

Score-Based Generative Models

While diffusion models are focused on learning forward noising and reverse denoising processes, score-based generative models take on another perspective of the same underlying framework.

Why use scores?

The problem with modelling probability distributions as mentioned above, is that they must be non-negative and the probabilities must sum to one.

• we use the exponential function to make sure the function is non-negeative
• we divide by Z to normalise the total probability to 1

However, Z(θ) requires integrating over all possible x, which is intractable in high dimensions.

Instead of modelling probability directly, score-based models model the gradient of the log probability:

Taking the gradient with respect to x:

The intractable constant Z(θ) disappears because it does not depend on x. This allows us to model the shape of the data distribution without needing to normalise it.

Stanford CS236: Deep Generative Models | 2023 | Lecture 13 — Score Based Models

As we use the gradient of the log probability as the score, we get a vector field with arrows that tell us the direction and magnitude of the steepest ascent.

• The arrows point toward the nearest ‘manifold’ where the data lives
• The arrows are longer in low-density regions and become zero exactly at the peaks of the distribution

As the model learns to predict the score, the Fisher divergence is used as the objective to minimise the difference between the score functions of two distributions.

As the true score function s(x) is unknown, Hyvärinen rewrote the intractable Fisher divergence equation into an objective that can be computed directly from data samples using the Hessian matrix:

However, computing the trace of the Hessian is not very scalable for high-dimensional data such as image as the Hessian is a d x d matrix of second order derivatives.

Hessian Matrix — Wikipedia

Sliced Score Matching

To make the computation more scalable, instead of directly comparing the full high-dimensional score vectors, sliced score matching compares their one-dimensional projections along randomly sampled directions.

This is the key idea: if the vector fields are the same, they should also match if you project them.

SSM ‘slices’ the high-dimensional space using random vectors, and projects the score onto this vector. If the projected scores match across all possible random directions v, the underlying multi-dimensional score vectors are mathematically guaranteed to be identical.

The sliced score matching objective is derived from the Sliced Fisher Divergence, which measures the difference between the projected score functions.

• v^T sθ(x) is the projection of the score on to the random vector v

Langevin Dynamics

Once the model has learned the score function, we can use it to refine random noise into realistic samples.

Langevin Markov Chain Monte Carlo (MCMC) is an iterative sampling method that uses the score function to guide samples towards regions of high probability density.

• xi is the current position of the Markov chain
• z is random Gaussian noise drawn from N(0, I)

Manifold Hypothesis Problem

Score matching was a breakthrough which helped to reframe generative modelling, but had issues initially as well:

In the Fisher divergence, the difference is weighted by p(x).

As real data distributions are highly concentrated, this means that most of the space has near-zero probability and the gradients are poorly defined in empty regions, making score estimates unstable.

Stanford CS236: Deep Generative Models | 2023 | Lecture 13 — Score Based Models

Denoising Score Matching

To address this problem, score-based generative models introduce noise into the data and learn the score of the resulting noisy distributions instead.

The addition of noise spreads out the probability mass, creating a smoother distribution whose score is easier to estimate. It also makes the calculations more tractable as the trace of the Hessian matrix is no longer required.

Noise is added via x̃ = x + ε , where ε ~ N(0, σ²I). The conditional probability density qσ(x̃|x) is a multivariate Gaussian centered at x.

To find the score function, we take the logarithm and compute the gradient with respect to x̃.

As the first term is a constant with respect to x̃, its derivative is 0.

As qσ(x̃|x) is a simple Gaussian, its score function can be calculated analytically with basic calculus.

• x : a clean, uncorrupted data point sampled from the training dataset
• x̃ : a corrupted, noise data point generated by adding the noise to the clean data point x

The noise ε points away from the clean data point x to the noisy point x̃. The negative noise vector -ε points directly back to the high-density area where the clean data lives.

By optimising this across the entire dataset, the network learns a continuous vector field. At any point, the network’s output vector points in the direction of the nearest valid data configuration.

However, adding noise itself turns the network into an approximation.

• If the noise level is too small, manifold hypothesis problem remains.
• If the noise level is too large, the distribution becomes overly smooth and the model loses information about the fine-grained structure of the data.

To address this tradeoff, score-based generative models are trained across multiple noise scales. The model is trained to estimate the score function at every noise level: σ1 > σ2 > … > σL

Stanford CS236: Deep Generative Models I 2023 I Lecture 14 — Energy Based Models

The objective function becomes a weighted combination of denoising score matching losses:

• λ is a hyperparameter that decides how much weight we put on each component of the loss

Stochastic Differential Equations

In diffusion models, noise is added in a finite number of timesteps.

What happens if we consider different noise levels inbetween? If we continuously interpolate between noise scales, we obtain an infinite family of probability distributions perturbed with increasingly large amounts of noise.

Instead of indexing these distributions using discrete timesteps t1, t2, …, tT, we now index them by t, where t is a continuous random variable.

Stanford CS236: Deep Generative Models I 2023 I Lecture 14 — Energy Based Models

Forward SDE The process of gradually transforming a data sample x into pure noise over time can be modelled as a continuous-time stochastic process using a forward stochastic differential equation (SDE).

• f(x, t)dt is the drift, indicating movement due to a deterministic force
• g(t)dw is the noise, indicating random fluctuations that could occur in the environment
• g(t) is the noise controls the amount of noise injected at time t

Score-based Diffusion Models | Generative AI Animated

Reverse SDE Any forward SDE has a corresponding reverse-time SDE. Similarly to the backward process for diffusion models, running this reverse SDE backward from t = T to t = 0 allows us to sample from the clean data distribution.

• ∇ log p_t(x) is the score function of the continuous data distribution at time t

To simulate this reverse process, we only need to know the score function ∇ log p_t(x) which is approximated using a score-based model trained via multi-scale weighted Denoising Score Matching.

Probability Flow ODE Every stochastic reverse SDE shares the same marginal probability densities as a deterministic Ordinary Differential Equation (ODE).

Unlike the reverse SDE, there is no random noise term present, making the trajectory completely deterministic. The ODE defines an invertible continuous transformation between the Gaussian (noise) distribution and the complex data distribution. Each initial noise sample maps to exactly one generated sample.

As a result, the probability flow ODE can be interpreted as a type of continuous normalising flow, a flow model which is trained by score matching instead of maximum likelihood estimation.

References

[1] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840–6851.

[2] Outlier. (2022, Jun 6). Diffusion Models | Paper Explanation | Math Explained. YouTube. https://www.youtube.com/watch?v=HoKDTa5jHvg

[3] Outlier. (2024). Diffusion Models From Scratch | Score-Based Generative Models Explained | Math Explained. YouTube. https://www.youtube.com/watch?v=B4oHJpEJBAA

[4] Weng, Lilian. (Jul 2021). What are diffusion models? Lil’Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

[5] Deepia (2025, Jun 26). Score-based Diffusion Models | Generative AI Animated. YouTube. https://www.youtube.com/watch?v=lUljxdkolK8

[6] Yang Song (2021, May 5). Generative Modeling by Estimating Gradients of the Data Distribution. https://yang-song.net/blog/2021/score/