I Built a Face Generator from Pure Noise Here’s Everything I Learned

How Denoising Diffusion Probabilistic Models actually work, and how to build one from scratch using PyTorch

There’s something almost philosophically beautiful about diffusion models. You start with pure, meaningless Gaussian noise — the most disordered thing imaginable — and through a series of small, learned denoising steps, a human face materializes. Not a template face. Not a morph. A new face that never existed before.

For Assignment 4 of my Generative AI course, I had to build exactly this: a complete DDPM pipeline from scratch — forward process, U-Net backbone, training loop, DDIM sampling, quantitative evaluation, and a Gradio demo — all trained on the CelebA-HQ dataset.

This is a writeup of everything I built, every design decision I made, and the intuitions I picked up along the way.

The Big Idea: What Is a Diffusion Model?

Before a single line of code, let’s get the mental model right.

A diffusion model has two processes:

The Forward Process is simple and fixed — no learning involved. You take a clean image x₀ and progressively add Gaussian noise over T timesteps until, at x_T, you have pure noise. This process is Markovian: each step only depends on the previous one.

The Reverse Process is where learning happens. A neural network (in our case, a U-Net) learns to predict the noise that was added at each step. At inference time, you start from x_T (random noise) and iteratively denoise, running the process in reverse — recovering a clean image.

The training objective is elegant: at each step, add noise to an image, then train the network to predict exactly that noise. That’s it. The loss is just MSE between the actual noise ε and the predicted noise ε̂_θ:

L = E[||ε − ε̂_θ(x_t, t)||²]

Simple on paper. Deceptively powerful in practice.

Step 1 — The Noise Schedule

Not all noise is created equal. How you schedule the noise across T timesteps dramatically affects training stability and sample quality.

Linear Schedule (the baseline)

The simplest approach: linearly interpolate β_t from β_start = 1e-4 to β_end = 0.02.

betas = torch.linspace(beta_start, beta_end, T)

Cosine Schedule (the upgrade)

The original DDPM paper used a linear schedule, but OpenAI’s improved DDPM paper showed that a cosine schedule is significantly better. The linear schedule destroys too much signal too quickly — by timestep 100, you’ve already added so much noise that the image is barely recognizable. The cosine schedule is gentler in the early steps and more aggressive later.

t = torch.linspace(0, T, steps) / T
f_t = torch.cos((t + 0.008) / 1.008 * math.pi / 2) ** 2
alphas_bar_raw = f_t / f_t[0]
betas = 1 - (alphas_bar_raw[1:] / alphas_bar_raw[:-1])
betas = betas.clamp(min=1e-5, max=0.9999)

The 0.008 offset prevents β_t from being too small near t=0, avoiding numerical instability. This schedule keeps more signal intact in the early timesteps, which gives the model better gradient signal to learn from.

I used T = 400 timesteps with the cosine schedule for all experiments.

Step 2 — The Dataset

The model trains on CelebA-HQ — a curated, high-quality version of CelebFaces Attributes — resized to 128×128. I loaded it from Kaggle with standard augmentations:

transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(0.1, 0.1, 0.1),
    transforms.ToTensor(),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])

The normalization maps pixel values to [-1, 1], which matches the range of our final tanh-style output. Color jitter adds slight variation in brightness, contrast, and saturation — simple but effective augmentation.

Batch size of 16 with pin_memory=True and drop_last=True (so every batch is the same size — important for stability).

Step 3 — The U-Net Architecture

The U-Net is the brain of the whole operation. Its job: given a noisy image x_t and a timestep t, predict the noise ε that was added.

Why U-Net?

U-Nets are encoder-decoder networks with skip connections between corresponding encoder and decoder layers. The encoder progressively downsamples to capture global context; the decoder upsamples back to full resolution. Skip connections preserve fine-grained spatial details. For image generation, this combination is ideal.

Timestep Conditioning — Sinusoidal Embeddings

The network needs to know what timestep it’s operating at. We encode t as a sinusoidal positional embedding (borrowed from Transformers):

class SinusoidalPositionEmbeddings(nn.Module):
    def forward(self, t):
        half_dim = self.dim // 2
        embeddings = math.log(10000) / (half_dim - 1)
        embeddings = torch.exp(torch.arange(half_dim) * -embeddings)
        embeddings = t[:, None].float() * embeddings[None, :]
        return torch.cat([embeddings.sin(), embeddings.cos()], dim=-1)

This embedding is then passed through a small MLP to get time_emb, which is injected into every ResidualBlock. High-frequency components capture fine-grained timestep differences; low-frequency components handle coarse structure.

Residual Blocks with Adaptive Normalization

Each ResidualBlock applies the time embedding via scale-shift normalization (also called AdaGN — Adaptive Group Normalization):

scale, shift = self.time_mlp(time_emb).chunk(2, dim=1)
h = h * (scale + 1) + shift

This is more expressive than simply adding the time embedding. The network learns to rescale feature maps differently at each timestep — effectively learning a different normalization for each noise level.

Memory-Efficient Attention

Attention is placed at the deepest level of the U-Net (where feature maps are smallest, keeping memory tractable). I used PyTorch’s built-in F.scaled_dot_product_attention with flash attention enabled:

with torch.backends.cuda.sdp_kernel(
    enable_flash=True,
    enable_math=True,
    enable_mem_efficient=True
):
    out = F.scaled_dot_product_attention(q, k, v, scale=self.scale)

This avoids materializing the full N×N attention matrix, making it 2–3× more memory efficient than naive attention — crucial when training on a T4 GPU.

Channel Structure

Base channels: 64
Channel multipliers: (1, 2, 4)
Actual channels per level: [64, 128, 256]

So the encoder goes: 3 → 64 → 128 → 256 (bottleneck), and the decoder mirrors it back to 3. Total parameters: roughly 28M.

Parameters: ~28,000,000 (~107 MB)

A quick sanity check before training:

xd = torch.randn(2, 3, 128, 128).to(device)
td = torch.randint(0, T, (2,)).to(device)
od = model(xd, td)
# xd.shape → od.shape: (2,3,128,128) → (2,3,128,128) ✓

Step 4 — Training

Optimizer and Scheduler

AdamW with weight decay (1e-4) and a learning rate of 2e-4. The weight decay acts as L2 regularization, helping prevent overfitting on the texture details of face images.

Cosine Annealing LR — the learning rate follows a cosine curve from lr_max = 2e-4 down to lr_min = 2e-6 over 50 epochs. This gentle cooldown in late training helps the model fine-tune without overshooting.

Mixed Precision Training

Training at float16 (AMP) cuts memory usage in half and speeds up training roughly 2× on Tensor Core GPUs like the T4:

scaler = torch.amp.GradScaler('cuda', enabled=True)

with torch.amp.autocast('cuda', enabled=True):
    pred_noise = model(xt, t)
    loss = criterion(pred_noise, noise)

scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
scaler.step(optimizer)
scaler.update()

The gradient scaler dynamically adjusts the loss scale to prevent underflow in float16 gradients — a common pitfall when training with AMP.

Gradient Clipping

clip_grad_norm_ with max_norm=1.0 prevents exploding gradients. Diffusion models are trained over thousands of iterations on noisy images — gradient spikes can silently corrupt weights if unchecked.

Exponential Moving Average (EMA)

EMA maintains a shadow copy of the model weights, updated after every training step:

ema_p.data.mul_(decay).add_(model_p.data, alpha=1 - decay)

With decay = 0.9999, the EMA model is a very slow-moving average of recent weight checkpoints. The result? EMA-sampled images are noticeably sharper and more stable than images from the raw training model. This is because the EMA model averages out the noise in gradient descent, converging to a smoother region of parameter space.

Always use the EMA model at inference time.

Early Stopping & Checkpointing

Training saves the best model whenever loss improves, and can resume from checkpoint if interrupted. Early stopping with patience=10 prevents wasteful computation if the model has converged.

Step 5 — Sampling

DDPM Sampling (Full T Steps)

The standard reverse process runs all T timesteps:

x_T ~ N(0, I)
for t = T-1 down to 0:
    ε̂ = model(x_t, t)
    x_{t-1} = (1/√α_t) * (x_t − β_t/√(1-ᾱ_t) * ε̂) + σ_t * z

where z ~ N(0, I) (added at all steps except t=0) and σ_t = √posterior_variance_t.

This works, but it’s slow — 400 denoising steps per image.

DDIM Sampling (50 Steps) — The Speed Unlock

Denoising Diffusion Implicit Models (Song et al., 2020) showed that you can skip most of the timesteps by reformulating the reverse process as deterministic (when η=0):

pred_x0 = (x - (1 - alpha_bar).sqrt() * pred_noise) / alpha_bar.sqrt()
pred_x0 = pred_x0.clamp(-1, 1)
dir_xt = (1 - alpha_bar_prev).sqrt() * pred_noise
x = alpha_bar_prev.sqrt() * pred_x0 + dir_xt

With ddim_steps=50 on a T=400 model, we get 8× speedup with comparable sample quality. At η=0, sampling is deterministic — same seed, same face. Increasing η towards 1.0 reintroduces stochasticity and increases diversity at the cost of sharpness.

I used DDIM during training evaluation (every 10 epochs) for fast preview, and DDPM for the final reconstruction comparisons.

Step 6 — Evaluation

Qualitative “it looks good” isn’t enough. I computed two metrics on 5 reconstructed image pairs (original vs. reconstructed after adding noise at t=250 and denoising back):

Metric What It Measures Score Range PSNR Pixel-level fidelity (higher = better) dB, higher is better SSIM Perceptual structural similarity (higher = better) [0, 1]

psnr_metric = PeakSignalNoiseRatio(data_range=1.0)
ssim_metric = StructuralSimilarityIndexMeasure(data_range=1.0)

Higher PSNR means the pixel values are closer between original and reconstruction. SSIM additionally captures structural similarity — texture, edges, contrast — which correlates better with human perception.

One interesting note: reconstruction at t=250 is not about pixel-perfect recovery (the model generates plausible completions, not exact copies). So SSIM is more meaningful here than PSNR — the structure should be preserved even if exact pixel values drift.

Step 7 — The Gradio Demo

The deployment layer is a Gradio app with two tabs:

Tab 1: Generate from Noise

• Pure Gaussian noise → denoised face via DDIM
• Sliders for: number of images, DDIM steps, eta (stochasticity)
• Shows denoising intermediate steps side-by-side

Tab 2: Reconstruct Your Image

• Upload any face image
• Control the noise level t (higher = more destroyed, more creative reconstruction)
• Returns: original, noisy version, and reconstructed output

demo.launch(share=True, debug=False)

The share=True flag creates a public tunneled URL — useful for demos without deploying to HuggingFace Spaces.

What I Learned (The Actually Useful Stuff)

1. Cosine schedule > linear schedule, always. The linear schedule destroys too much signal too early. Cosine is smoother and produces better samples with the same number of training steps.

2. EMA is not optional. Without EMA, generated samples are visibly noisy and inconsistent. With decay=0.9999, the difference is dramatic — smoother, more coherent faces.

3. DDIM is the practical choice. Going from 400 steps to 50 steps with near-identical quality is not a small win — it’s an 8× inference speedup. For any production application, you’d use DDIM (or its successor DEIS, DPM-Solver, etc.).

4. Mixed precision training is worth the complexity. AMP + GradScaler is a 15-minute setup that gives you 2× speed and 2× memory efficiency. There’s no good reason not to use it on modern GPUs.

5. The U-Net skip connections are doing a lot of work. Early in training, the model learns coarse structure via the bottleneck. The skip connections refine fine detail as training continues. Removing them tanks sample quality immediately.

6. Attention placement matters. Putting attention only at the deepest level (smallest feature maps) keeps memory usage tractable without significantly hurting quality. Full-resolution attention on 128×128 would OOM immediately on a T4.

Architecture at a Glance

Input (3×128×128)
    │
    ▼
Init Conv → [64 channels]
    │
    ├── ResBlock → ResBlock → Downsample     [64 → 128]
    │
    ├── ResBlock → ResBlock → Downsample     [128 → 256]
    │
    ├── ResBlock → ResBlock → AttentionBlock [256, bottleneck]
    │
    ├── Bottleneck: ResBlock → Attention → ResBlock
    │
    ├── Upsample → ResBlock → ResBlock → AttentionBlock [256 → 128]
    │
    ├── Upsample → ResBlock → ResBlock                  [128 → 64]
    │
    ▼
GroupNorm → SiLU → Conv(1×1) → Output (3×128×128)

Every ResBlock receives the sinusoidal time embedding via scale-shift conditioning. Skip connections flow from every encoder level to its mirror decoder level.

The Philosophical Bit

There’s something genuinely strange about watching this work. A diffusion model doesn’t have a latent space in the traditional sense. It doesn’t compress an image to a vector and decode it. Instead, it learns the gradient of the data distribution — how to move slightly more towards “looks like a face” at each denoising step.

The forward process is physics (diffusion). The reverse process is a learned approximation of physics running backwards in time. The fact that this produces photorealistic faces from random noise is, once you sit with it, quite remarkable.

This assignment gave me a working implementation of one of the most important architectures in modern generative AI — the same family of models that powers Stable Diffusion, DALL·E, Midjourney, and Sora. The core ideas here — score matching, denoising objectives, U-Net conditioning — are alive in all of them.

If you’re going to understand modern generative AI from first principles, DDPMs are the right place to start.

Code is available on GitHub. Dataset: CelebA-HQ (Kaggle). Training hardware: Kaggle T4 x2 GPU. Framework: PyTorch 2.x.