Building a Diffusion Model from Scratch: How We Implemented DDPM for High-Resolution Image Generation

No pretrained pipelines. No HuggingFace Diffusers. Just PyTorch, math, and a lot of patience.

Introduction

Diffusion models have quietly taken over the generative AI world. Stable Diffusion, DALL·E, Imagen — they all share a common ancestor: the Denoising Diffusion Probabilistic Model (DDPM), introduced by Ho et al. in 2020.

For our Generative AI course assignment at NUCES (FAST), my teammate Hadia Javed and I built a complete DDPM from scratch using base PyTorch — no pretrained pipelines, no shortcuts. We trained it on the CelebA-HQ dataset and got it generating faces from pure Gaussian noise.

This article is a full walkthrough of everything we built, every decision we made, and every mistake we learned from.

What Is a Diffusion Model?

The core idea is beautifully simple:

1. Forward process — Gradually add Gaussian noise to an image over T timesteps until it looks like pure static.
2. Reverse process — Train a neural network to undo that noise, one step at a time.

At inference time, you start from pure noise and run the reverse process — and the network hallucinates a realistic image from nothing.

Mathematically, the forward process is:

q(xₜ | xₜ₋₁) = N(xₜ; √(1-βₜ) xₜ₋₁, βₜI)

Where βₜ is a noise schedule that controls how much noise is added at each step. Using the "reparameterization trick," you can jump directly to any noisy timestep t from the original image x₀:

xₜ = √(ᾱₜ) · x₀ + √(1 - ᾱₜ) · ε,  where ε ~ N(0, I)

This makes training efficient — you don’t need to iteratively noise an image; you can sample any timestep in one shot.

Our Implementation

1. The Noise Schedule

We implemented both linear and cosine schedules. The cosine schedule (Nichol & Dhariwal, 2021) is significantly better — it avoids destroying too much image structure in the early timesteps.

# Cosine schedule
x = torch.linspace(0, T, steps)
alphas_cumprod = torch.cos(((x / T) + s) / (1 + s) * math.pi / 2) ** 2
alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
betas = torch.clamp(betas, 0.0001, 0.9999)

We used T = 300 timesteps — enough for quality results while keeping training time manageable on Kaggle’s T4 GPUs.

2. The U-Net Backbone

The denoising network is a U-Net — an encoder-decoder architecture with skip connections that let the network preserve fine spatial detail while also understanding global structure.

Our channel progression: 64 → 128 → 256

Key components we implemented:

Residual Blocks — The backbone of each resolution level. They apply two convolutions with GroupNorm and SiLU activation, with a shortcut connection:

class ResidualBlock(nn.Module):
    def forward(self, x, t_emb):
        h = self.conv1(F.silu(self.norm1(x)))
        h = h + self.time_proj(F.silu(t_emb))[:, :, None, None]
        h = self.conv2(self.dropout(F.silu(self.norm2(h))))
        return h + self.res_conv(x)

Sinusoidal Time Embeddings — Borrowed from transformers, these encode the timestep t as a continuous vector so the network knows "how noisy" the input is:

freqs = torch.exp(-math.log(10000) * torch.arange(half) / (half - 1))
emb = torch.cat([torch.sin(t * freqs), torch.cos(t * freqs)], dim=-1)

Self-Attention at the Bottleneck — We added a self-attention block at the lowest spatial resolution, allowing the model to reason about global structure.

Downsampling / Upsampling — Strided convolutions for down, nearest-neighbor interpolation + conv for up (smoother than transposed convolutions).

3. Training Setup

Loss:       MSE between predicted noise and actual noise
Optimizer:  AdamW (lr=2e-4, weight_decay=1e-4)
Scheduler:  Cosine Annealing
Precision:  Mixed (torch.cuda.amp)
Batch size: 16
Epochs:     30
GPUs:       2× T4 via DataParallel

The training objective is simple: the network predicts the noise ε added at timestep t, and we minimize:

L = ||ε - ε_θ(xₜ, t)||²

Mixed precision training (autocast + GradScaler) cut our memory usage significantly and let us run without hitting Kaggle's GPU limits.

4. The Reverse Process (Sampling)

At inference time, we start with x_T ~ N(0, I) and iteratively denoise:

for t in reversed(range(T)):
    pred_noise = model(x, t_batch)
    x0_pred = sqrt_recip * x - sqrt_recm1 * pred_noise
    x0_pred = torch.clamp(x0_pred, -1, 1)
    mean = c1 * x0_pred + c2 * x
    x = mean + (0.5 * log_var).exp() * noise  # add small noise (except last step)

5. Image Reconstruction

One of the most interesting tasks: given a target image, can we reconstruct it using the diffusion process?

We do this by:

1. Adding noise to the target up to timestep t_start = 200
2. Running the reverse process from there back to t = 0

This is a form of “image-to-image” diffusion — the output should resemble the target without being an exact copy.

We evaluated reconstruction quality using PSNR and SSIM metrics from torchmetrics.

6. The Gradio App

We wrapped the model in a Gradio app with a slider to control how many intermediate denoising steps to display. Users can watch the model paint an image from noise in real time.

demo = gr.Interface(
    fn=gradio_generate,
    inputs=[gr.Slider(3, 10, value=5, label='Denoising steps to show')],
    outputs=[gr.Image(label='Generated'), gr.Image(label='Denoising Steps')]
)
demo.launch(share=True)

Results

After 30 epochs on 20,000 CelebA-HQ images at 128×128:

• The model generates recognizable face-like structures from pure noise
• Forward and reverse diffusion visualizations clearly show the noising/denoising process
• PSNR and SSIM scores confirm measurable reconstruction quality

The loss curve showed stable, monotonically decreasing training loss — a good sign that the network was learning and not diverging.

Key Lessons Learned

Cosine schedule > linear schedule. The linear schedule destroys image structure too aggressively in early timesteps. Cosine is smoother and produces better results.

GroupNorm > BatchNorm for diffusion. Batch statistics fluctuate heavily when batch size is small. GroupNorm is stable regardless of batch size.

Mixed precision is not optional on Kaggle. Without autocast, we couldn't fit a reasonable batch size in T4 memory.

Gradient clipping matters. Without it, training occasionally spiked and diverged. clip_grad_norm_(model.parameters(), 1.0) kept everything stable.

T doesn’t need to be 1000. The original DDPM used 1000 timesteps. We got solid results at 300, which made both training and sampling 3× faster.

What We’d Do Differently

• Train at 256×256 for sharper outputs (needs more compute)
• Implement DDIM sampling for faster inference (50 steps instead of 300)
• Add class conditioning to control what kind of image gets generated
• Experiment with EMA (Exponential Moving Average) of model weights for more stable generation

Conclusion

Building a diffusion model from scratch is one of the best ways to understand what’s actually happening inside tools like Stable Diffusion. The math is elegant, the implementation is challenging in just the right ways, and watching your model slowly learn to paint faces from noise is genuinely satisfying.

Huge thanks to my teammate Hadia Javed for her collaboration and for making this project what it is. 🙏

All code is implemented in base PyTorch. No pretrained diffusion pipelines were used.

Tags: #MachineLearning #DiffusionModels #DDPM #PyTorch #GenerativeAI #DeepLearning #ComputerVision