Why I Moved from DCGAN to WGAN-GP: Building More Stable GAN Training

Generative Adversarial Networks are among the most fascinating ideas in deep learning because they attempt to teach neural networks how to create entirely new data from randomness itself. But while GANs often look elegant in theory, training them in practice can quickly become unpredictable, unstable, and surprisingly difficult. While experimenting with large-scale image generation, I gradually realized that many of the challenges in GANs go far beyond simply designing better neural network architectures. Some of the most interesting questions started appearing only after the actual training began.

Table of Contents

1. Understanding GANs Again Briefly
2. Why DCGAN Was Introduced
3. Problems I Encountered During DCGAN Training, Planned Mitigations, and Remaining Drawbacks
4. What is WGAN?
5. Choosing the Right Distance Metric for WGANs?
6. Introduction to the Critic in WGAN
7. How WGAN works?
8. Lipschitz Constraint in WGAN
9. What is WGAN-GP and how is it different from WGAN?
10. How WGAN-GP works?
11. Why WGAN-GP Uses Interpolated Images for Gradient Penalty?
12. Why Batch Normalization Is Often Removed from the Critic in WGAN-GP?
13. Final Thoughts

1. Understanding GANs Again Briefly

A GAN consists of two neural networks competing against each other.

The generator receives random noise as input and attempts to transform it into realistic images.

The discriminator receives both:

• real images from the dataset
• fake images generated by the generator

Its goal is to correctly classify them.

The adversarial objective can be represented as:

Over time:

• the generator becomes better at fooling the discriminator
• the discriminator becomes better at detecting fake images

Ideally, the generator eventually learns the true data distribution.

In practice, things are far more unstable.

2. Why DCGAN Was Introduced

The original GAN architecture struggled with image generation quality.

DCGAN introduced:

• convolutional layers
• transposed convolutions
• batch normalization
• deeper feature extraction

This significantly improved image generation.

Generator pipeline:

Discriminator pipeline:

DCGAN became one of the foundational architectures for image generation.

But even DCGAN remained highly unstable.

3. Problems I Encountered During DCGAN Training, Planned Mitigations, and Remaining Drawbacks

While training a DCGAN on NVIDIA’s face dataset containing nearly 70,000 images, I encountered several practical and optimization-related challenges. The training process became increasingly difficult to manage efficiently because each epoch required processing more than a thousand batches, making experimentation and evaluation extremely slow. Infrastructure limitations on Google Colab further complicated the workflow, as long-running sessions could terminate unexpectedly and GPU usage limits could be exhausted during extended training runs. To improve recovery and experimentation, I implemented batch-wise checkpointing so that training progress could be restored from intermediate stages whenever interruptions occurred.

These challenges gradually pushed me to rethink the overall training workflow. Instead of relying entirely on conventional epoch-based training, I started exploring chunk-based training strategies in which the dataset could be divided into smaller subsets for faster experimentation, more frequent checkpointing, and earlier inspection of generated outputs. I also considered globally shuffled chunk sampling and continuous monitoring of generated images using fixed latent vectors to make comparisons across different training stages more consistent.

However, even after planning these workflow-level improvements, I realized that they did not fundamentally solve the deeper instability present in traditional GAN optimization itself. Problems such as unstable gradients, oscillation, and mode collapse could still emerge because the core issue appeared to originate from the adversarial objective function used in conventional GANs. This realization eventually motivated me to explore WGAN and WGAN-GP.

4. What is WGAN?

Wasserstein Generative Adversarial Network (WGAN) is an improved version of the traditional Generative Adversarial Network designed to make GAN training more stable and reliable.

In a traditional GAN, the discriminator mainly tries to classify images as either real or fake. However, this binary decision-making process often causes unstable training, weak gradients, and problems such as mode collapse.

WGAN changes this approach completely. Instead of simply deciding whether an image is real or fake, WGAN attempts to measure how different the generated data distribution is from the real data distribution. WGAN treats it as a distribution-matching problem.

Conceptually, WGAN asks:

“How much effort is required to transform the generated distribution into the real distribution?”

This provides the generator with smoother and more meaningful feedback during training.

As a result, WGAN improves:

• training stability
• gradient flow
• convergence behavior
• diversity of generated outputs

The core principle of WGAN is based on the idea of measuring the “distance” between the real data distribution and the generated data distribution.

5. Choosing the Right Distance Metric for WGANs?

When training a GAN, the fundamental objective is to make the generated distribution Pg as close as possible to the real data distribution Pr.

Several distance metrics are:

1. Total Variation (TV) Distance

The Total Variation distance is defined as:

Here sup means largest possible value.

Intuitively, it measures the maximum probability difference between two distributions over all possible regions.

Total Variation (TV) distance is not very effective for image generation because image distributions exist in extremely high-dimensional spaces and often have very little overlap during early training. As a result, TV distance quickly saturates and behaves almost like:

completely different or not different

without providing smooth information about how close the generated distribution is to the real distribution. This causes weak or uninformative gradients, making GAN optimization unstable.

2. Kullback–Leibler (KL) Divergence

KL divergence is defined as:

Here we use dμ(x) is used instead of dx because GAN distributions exist in very high-dimensional and general probability spaces where ordinary calculus integration using dx is not always mathematically sufficient.

KL divergence measures how much information is lost when Pg approximates Pr​.

However, KL divergence has major problems for GAN training.

If:

Pg(x)=0 but Pr(x)>0

then:

KL divergence → ∞

This becomes extremely unstable during optimization.

KL divergence is also asymmetric:

KL divergence is asymmetric because the information loss when approximating Pr using Pg​ is generally different from approximating Pg​ using Pr.

3. Jensen–Shannon (JS) Divergence

Traditional GANs effectively optimize Jensen–Shannon divergence.

It is defined as:

where:

JS divergence is symmetric and more stable than KL divergence.

However, it still suffers from a critical problem.

In very high-dimensional spaces such as images:

real and generated distributions often barely overlap initially

When this happens, JS divergence becomes nearly constant.

As a result:

gradients become extremely weak

and the generator stops learning effectively.

This is one of the major reasons why traditional GAN training becomes unstable.

4. Earth Mover’s Distance (Wasserstein Distance)

It is defined as:

Here:

• inf means the smallest possible value
• γ represents a transport plan describing how probability mass moves from generated samples to real samples
• Π(Pr,Pg) represents all valid transport plans between the two distributions
• x and y represent points from the real and generated distributions respectively
• ∥x−y∥ measures the distance the probability mass must move

Overall, Wasserstein Distance searches for the transport plan with the minimum average movement cost required to transform the generated distribution into the real distribution.

Unlike traditional divergence metrics, Wasserstein Distance changes smoothly even when the two distributions barely overlap, which provides meaningful gradients and significantly improves GAN training stability.

Therefore we use this metric in WGAN.

6. Introduction to the Critic in WGAN

In traditional GANs, the discriminator acts as a binary classifier that predicts whether an image is real or fake:

Real → 1 Fake → 0

However, when the discriminator becomes too accurate, its outputs saturate near 0 or 1, causing the generator to receive weak gradients.

WGAN replaces the discriminator with a critic, which does not perform binary classification. Instead, it assigns continuous scores to estimate how different the generated distribution is from the real distribution. Unlike the discriminator, the critic does not use a sigmoid activation function and outputs unrestricted real-valued scores.

Conceptually:

Discriminator: “Is this image real or fake?”

Critic: “How far is the generated distribution from the real distribution?”

The critic aims to:

maximize scores for real images minimize scores for generated images

while the generator aims to increase critic scores for fake images.

7. How WGAN works?

The original Wasserstein Distance is defined using optimal transport theory as:

This formulation measures the minimum transportation cost required to transform the generated distribution into the real data distribution.

Although mathematically elegant, this formulation is computationally intractable for neural network optimization because it requires searching over all possible transport plans γ.

To make Wasserstein Distance usable for GAN training, WGAN applies the Kantorovich–Rubinstein duality, which converts the original transport formulation into a neural-network-friendly optimization objective.

This produces the practical WGAN critic objective:

Instead of explicitly computing transport plans, the critic now estimates the Wasserstein Distance by assigning higher scores to real samples and lower scores to generated samples.

8. Lipschitz Constraint in WGAN

The Lipschitz constraint is a mathematical condition imposed on the critic in WGAN to ensure that Wasserstein Distance is computed correctly. A function f(x) is called K-Lipschitz continuous if:

In WGAN, the critic must satisfy the special case of 1-Lipschitz continuity, meaning the gradient norm of the critic should remain close to 1:

Intuitively, this ensures that small changes in input do not produce excessively large changes in output, keeping the critic smooth and stable.

To approximately enforce this constraint, the original WGAN introduced weight clipping, where critic weights were forced into a small fixed range after every optimization step:

Example:

The goal was to:

• prevent extremely sharp critic functions
• control gradient growth
• approximately maintain Lipschitz continuity

However, weight clipping created new problems:

• critic capacity became restricted
• gradients could vanish or explode
• optimization became unstable
• the critic often became too weak to learn complex distributions effectively

9. What is WGAN-GP and how is it different from WGAN?

WGAN-GP (Wasserstein GAN with Gradient Penalty) is an improved version of WGAN designed to stabilize training more effectively. Like WGAN, it uses:

• Wasserstein Distance
• a critic instead of a discriminator
• continuous distribution comparison instead of binary classification

The key difference is how the Lipschitz constraint is enforced. Instead of using weight clipping, WGAN-GP introduces a gradient penalty term in the critic loss:

where:

Here:

• λ controls penalty strength
• x represents interpolated samples between real and fake images:

where ϵ∈[0,1]. For example, ϵ=0.7 means:

70% real image 30% generated image

D(x̂) = critic output. The critic assigns a score to sample x̂.

∇x̂ D(x̂) = gradient of critic output with respect to input x̂. It measures: how rapidly the critic output changes when input changes slightly.

|| · ||₂ = L2 norm (gradient magnitude). This computes the size of the gradient vector. Intuitively: how strong the critic changes are

||∇x̂ D(x̂)||₂ — 1 = This checks: how far the gradient magnitude is from 1. Because WGAN requires: gradient norm ≈ 1 for 1-Lipschitz continuity.

( … )² = square penalty. Squaring: makes large violations penalized more strongly keeps penalty positive smooths optimization

WGAN:

“Force the critic weights to stay small.”

WGAN-GP:

“Allow flexible weights, but penalize unstable gradients.”

This produces:

• smoother optimization
• stronger critic learning
• more stable gradients
• reduced mode collapse
• improved convergence

10. How WGAN-GP Works?

Step 1 — Sample Real Images

The process begins by sampling a batch of real images from the real dataset:

x ~ Pr

These images represent the true data distribution Pr The critic should ideally assign: higher scores to these images because they belong to the real distribution.

Step 2 — Generate Fake Images

Random latent vectors are sampled:

z ~ p(z)

These latent vectors are passed into the generator:

x_fake = G(z)

The generator transforms random noise into synthetic images.

Initially, these fake images may look unrealistic because the generator has not learned the real distribution yet.

Step 3 — Create Interpolated Samples

WGAN-GP creates interpolated samples between:

• real images
• generated images

using:

x̂=ϵxreal​+(1−ϵ)xfake​

These interpolated samples are used to enforce the Lipschitz constraint smoothly between the two distributions.

Step 4 — Critic Evaluates Images

The critic receives:

• real samples
• fake samples
• interpolated samples

Unlike a traditional GAN discriminator, the critic does not output probabilities.

Instead, it outputs unrestricted real-valued scores:

Higher score → more real Lower score → more fake

The critic tries to:

maximize scores for real images minimize scores for generated images

Step 5 — Compute Wasserstein Distance Estimate

The critic computes:

This estimates the Wasserstein Distance between:

• real distribution
• generated distribution

If:

• critic assigns very high scores to real images
• critic assigns low scores to fake images

then the distance becomes large.

As the generator improves, this distance gradually decreases.

Step 6 — Compute Gradient Penalty

WGAN-GP then computes the gradient penalty:

This checks whether the critic gradients remain close to 1.

If gradients become:

• too large
• too small

the model adds a penalty.

This enforces:

• smooth critic behavior
• stable gradients
• 1-Lipschitz continuity

without using weight clipping.

Step 7 — Critic Training

The final critic loss becomes:

The critic updates its parameters to:

• better separate real and fake distributions
• estimate Wasserstein Distance more accurately
• maintain smooth gradients

In WGAN-GP, the critic is usually trained multiple times before updating the generator.

Typically:

Train Critic 5 times → Train Generator 1 time

This is done because the generator depends entirely on the critic for meaningful gradients. If the critic is weak, Wasserstein estimates become inaccurate and generator learning becomes unstable.

Training the critic multiple times allows it to:

• estimate distribution differences more accurately
• provide smoother gradients
• stabilize generator learning

Step 8 — Generator Training

After critic updates, the generator is trained.

Generator loss:

The generator tries to:

increase critic scores for fake images make fake images appear more realistic

If fake images start receiving higher critic scores:

generated distribution moves closer to real distribution

11.Why WGAN-GP Uses Interpolated Images for Gradient Penalty?

WGAN-GP uses interpolated images instead of only real images because the goal is to enforce the Lipschitz constraint in the region between the real and fake distributions, where the critic actually learns to distinguish them. If gradient penalty were applied only on real images, the critic’s gradients would be constrained only around real samples, leaving the space between real and generated images unconstrained and potentially unstable. By creating interpolated samples — random weighted combinations of real and fake images — the model forces the critic to maintain smooth, well-behaved gradients in the exact regions where fake samples move toward real ones during training. This helps prevent exploding gradients, stabilizes critic learning, and provides meaningful gradient signals to the generator for better image synthesis.

12. Why Batch Normalization Is Often Removed from the Critic in WGAN-GP?

In WGAN-GP, the gradient penalty is applied to individual interpolated samples:

For every interpolated image, WGAN-GP computes:

to measure how rapidly the critic output changes for that specific sample. The gradient penalty is therefore intended to enforce smooth gradient behavior independently for each sample.

However, Batch Normalization computes normalization statistics using the entire mini-batch. This means the critic output for one image becomes dependent on other images present in the batch.

As a result:

• gradients across samples become coupled
• per-sample gradient behavior is no longer independent
• the gradient penalty no longer accurately enforces the 1-Lipschitz constraint

Because of this, BatchNorm is often removed from the critic in WGAN-GP. Most implementations keep BatchNorm only inside the generator while using alternative normalization methods or no normalization inside the critic.

12. Final Thoughts

While experimenting with GANs, I realized that stable image generation is far more challenging than simply designing architectures. Many issues such as unstable gradients, mode collapse, oscillation, and difficult optimization only became apparent during actual training. Although approaches like checkpointing, chunk-based training, and interactive monitoring improved experimentation, they did not fundamentally solve the instability of traditional GAN optimization. This eventually motivated the transition toward WGAN and WGAN-GP, where image generation is treated as a distribution-matching problem using Wasserstein Distance and smoother critic-based optimization.