From DCGAN to WGAN-GP: What Actually Happens When You Train GANs

Objective

This project aimed to experimentally investigate GAN instability, specifically:

• Mode collapse
• Training imbalance
• Effectiveness of WGAN-GP as a fix

Experimental Setup

Dataset & Configuration

• Dataset: Pokémon sprites (~33K images)
• Image size: 64×64
• Latent vector: 100
• Batch size: 64
• Epochs: 100 (full training)
• Optimizer: Adam (lr = 0.0002, betas = 0.5, 0.999)

Experiment 1: DCGAN (Baseline)

Final Training Behavior (Epoch 100)

BatchD_lossG_loss00.00426.94561000.02347.61222000.01499.92373000.07896.92634000.00148.78945000.00369.7317

Key Observations

1. Discriminator Dominance

• D_loss ≈ 0.001–0.07
• Indicates near-perfect classification of real vs fake

Discriminator is too strong

2. Generator Struggling

• G_loss ≈ 6.9–9.9
• Extremely high → Generator failing to fool D

3. Persistent Instability (Even After 100 Epochs)

Despite long training:

• No convergence
• No stable equilibrium
• Losses remain extreme

Core Problem: Mode Collapse

Even after 100 epochs:

• Generator outputs low diversity
• Repeats similar patterns/images
• Fails to capture full data distribution

Why DCGAN Failed (Theoretical Link)

From the original GAN formulation:

min⁡Glog⁡(1−D(G(z)))\min_G \log(1 — D(G(z)))Gmin​log(1−D(G(z)))

When:

• D(x)→1D(x) \to 1D(x)→1
• D(G(z))→0D(G(z)) \to 0D(G(z))→0

Then: 👉 Gradients for Generator vanish 👉 Learning becomes ineffective

Insight

More epochs ≠ better GAN performance (100 epochs still failed to fix instability)

Experiment 2: WGAN-GP (Improved Model)

Final Training Behavior (Epoch 100)

BatchCritic LossGenerator LossGradient Penalty0–37.3941571.1215.43100–23.5941559.0022.8820038.2241589.5313.62300–46.9941581.9117.40400–52.0541880.0343.26500–155.7841673.1536.48

Key Observations

1. Critic Loss Behavior

• Ranges from +38 to -155
• Unlike BCE loss → not bounded
• Represents Wasserstein distance approximation

More meaningful gradients than DCGAN

2. Generator Loss Scale

• Extremely high (~41,000+)
• Expected in WGAN formulation (not directly comparable to BCE)

3. Gradient Penalty (Critical Component)

• Range: 13 → 43
• Enforces Lipschitz constraint

This is what stabilizes training

Important Insight (Very Strong Point)

Even though:

• Numerical values look “extreme”
• Losses fluctuate heavily

WGAN is more stable in practice because:

• Gradients do not vanish
• Generator keeps learning
• Mode collapse is reduced

Direct Comparison

AspectDCGANWGAN-GPLoss TypeBCEWassersteinD/C BehaviorOverconfidentBalanced criticG Loss6–10~41,000StabilityPoorImprovedMode CollapseSevereReducedGradient FlowVanishingStable

What Didn’t Go Well (Critical Reflection)

1. DCGAN Failure Was Expected

• Matches theory from GAN paper
• Not a bug → structural limitation

2. WGAN-GP Still Not Perfect

• Loss values highly volatile
• GP values sometimes high (~43)
• Training slower (~8 it/s vs 30 it/s)

3. Computational Cost

• DCGAN: ~30 it/s
• WGAN-GP: ~8 it/s

~4x slower

Final Takeaways

1. GAN Training is Fundamentally Unstable

Not due to implementation errors, but:

• Objective function design
• Adversarial dynamics

2. WGAN-GP Improves Learning Dynamics, Not Just Loss

It fixes:

• Gradient issues
• Mode collapse tendency

But:

• Does not make training trivial

3. Numbers Can Be Misleading

• Low loss ≠ good model (DCGAN)
• High loss ≠ bad model (WGAN)

👉 Visual outputs matter more

Final Insight

The real challenge in GANs is not building them — it’s making them learn correctly.

Github: https://github.com/abrar3652/Tackling-Mode-Collapse-in-Generative-Adversarial-Networks Live Link: https://dcgan-wgan.streamlit.app/