From Noise to Art: Building DCGAN, WGAN-GP, Pix2Pix, and CycleGAN from Scratch

A hands-on walkthrough of three GAN assignments — tackling mode collapse, image translation, and unpaired domain adaptation.

Introduction

Generative Adversarial Networks remain one of the most fascinating — and frustrating — areas of deep learning. For Assignment 3 of my Generative AI course (AI4009) at FAST-NUCES, we were tasked with implementing three distinct GAN architectures from scratch in PyTorch, each solving a different real-world problem. Here’s everything I learned.

— -

Question 1: Beating Mode Collapse with DCGAN and WGAN-GP

The problem with vanilla GANs

Mode collapse is arguably the most notorious failure mode in GAN training. Instead of learning the full data distribution, the generator “collapses” — it finds a small set of outputs that consistently fool the discriminator and keeps generating just those. The result? Repetitive, low-diversity images.

Baseline: DCGAN

We started with a standard Deep Convolutional GAN:

• 100-dimensional noise vector as input
• Generator: Transposed Conv → BatchNorm → ReLU → Tanh output
• Discriminator: Conv → LeakyReLU → Sigmoid output
• Loss: Binary Cross Entropy
• Trained on Pokemon Sprites and Anime Faces (64×64)

The DCGAN produced reasonable results but showed classic signs of instability — training losses oscillated, and generated samples lacked diversity.

Advanced: WGAN-GP

The Wasserstein GAN with Gradient Penalty addresses this directly. Key changes:

• Replace the discriminator with a Critic (no Sigmoid)
• Use Wasserstein loss instead of BCE
• Add Gradient Penalty (λ = 10) to enforce the Lipschitz constraint
• Run 5 critic updates per generator update

The effect was immediately noticeable — training was far more stable, loss curves were smoother, and the generated image diversity improved significantly.

Training setup on Kaggle T4×2

• Mixed precision (torch.cuda.amp) to save GPU memory
• Batch size: 64
• Adam optimizer, lr=0.0002, betas=(0.5, 0.999)
• Checkpoints every 5–10 epochs

— -

Question 2: Sketch to Reality with Pix2Pix

What is Pix2Pix?

Pix2Pix is a conditional GAN that learns a mapping between paired images — given a sketch, generate the corresponding realistic photo. We used two datasets: CUHK Face Sketch and Anime Sketch Colorization pairs.

Architecture highlights

Generator — U-Net:

• Encoder-decoder with skip connections
• Skip connections preserve spatial details (edges, structures) that would otherwise be lost in downsampling
• Input: sketch/grayscale image → Output: realistic/colored image

Discriminator — PatchGAN:

• Classifies 16×16 image patches as real or fake (not the whole image)
• Focuses on local texture realism
• Outputs a matrix of probabilities

Loss function

Total Loss = Adversarial Loss + λ × L1 Loss

The L1 reconstruction loss is crucial — it forces the generator to stay close to the ground truth pixel-by-pixel, while adversarial loss handles perceptual realism.

Evaluation

We used SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio) to quantitatively evaluate output quality beyond just visual inspection.

— -

Question 3: No Pairs Needed — CycleGAN for Domain Translation

Why CycleGAN?

Pix2Pix requires paired training data — for every sketch, you need its corresponding photo. CycleGAN removes this constraint entirely. Using two generators and two discriminators, it learns bidirectional translation (Sketch ↔ Photo) purely from unpaired images.

Architecture

• G_AB: Sketch → Photo (ResNet-based, 6 blocks)
• G_BA: Photo → Sketch
• D_A: Classifies sketch domain patches
• D_B: Classifies photo domain patches
• Image size: 128×128 (reduced for Kaggle memory)

Datasets: TU-Berlin, Sketchy, and Google QuickDraw.

The three losses that make CycleGAN work

1. Adversarial Loss — generators fool their respective discriminators
2. Cycle Consistency Loss — translating A→B→A should return the original A
3. Identity Loss — G_AB applied to a photo should return the same photo (color preservation)

The cycle consistency constraint is what makes the whole system stable without paired data. Without it, the generators could just ignore the input.

Training on Kaggle T4×2

• Batch size: 4–8 (CycleGAN is memory intensive — 4 networks)
• Mixed precision throughout
• Subset of each dataset used for feasibility

— -

Key takeaways

GANs reward patience and careful debugging. A few things I found most valuable:

1. Wasserstein loss genuinely makes GAN training more stable — the loss values are actually interpretable (lower = better), unlike BCE where oscillation is the norm.
2. Skip connections in U-Net are the reason Pix2Pix works so well for sketches — they’re not optional.
3. CycleGAN’s cycle consistency loss is elegant — it’s self-supervised regularization built into the architecture.
4. Mixed precision training on Kaggle T4×2 is almost mandatory for experiments at this scale.

GenerativeAI #GANs #DeepLearning #PyTorch #MachineLearning