Generative Adversarial Networks (GANs)

To see jupyter notebooks of this article and math formuls in latex click here

A generative adversarial network (GAN) is a class of machine learning frameworks and a prominent framework for approaching generative AI. Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics.

Generative modeling is an unsupervised learning task in machine learning that involves automatically discovering and learning the regularities or patterns in input data in such a way that the model can be used to generate or output new examples that plausibly could have been drawn from the original dataset.

Discriminative vs. Generative Modeling

In supervised learning, we may be interested in developing a model to predict a class label given an example of input variables. This predictive modeling task is called classification. Classification is also traditionally referred to as discriminative modeling.

Alternately, unsupervised models that summarize the distribution of input variables may be able to be used to create or generate new examples in the input distribution. As such, these types of models are referred to as generative models.

What Are Generative Adversarial Networks?

Generative adversarial networks are based on a game theoretic scenario in which the generator network must compete against an adversary. The generator network directly produces samples. Its adversary, the discriminator network, attempts to distinguish between samples drawn from thetraining data and samples drawn from the generator.

Generative modeling is an unsupervised learning problem, The two models, the generator and discriminator, are trained together. The generator generates a batch of samples, and these, along with real examples from the domain, are provided to the discriminator and classified as real or fake. The discriminator is then updated to get better at discriminating real and fake samples in the next round, and importantly, the generator is updated based on how well, or not, the generated samples fooled the discriminator.

Standard GAN loss function (min-max GAN loss)

Generator Loss

Generator wants to fool the Discriminator so in loss calculation we consider one matrix to calculate the loss so the BCE formular will be:

where the y^ is the last output of the Discriminator:

Discriminator Loss

one matrix to calculate the loss for real input and zeors matrix to calculate the loss for fake input which is the output of generator so the BCE formular will be:

When label is 1 (for real ones):

When label is 0 (for generated images)

where the y^ is the last output of the Discriminator:

Problems in GANs

• Mode collapse:

During the training, the generator may collapse to a setting where it always produces same outputs.

two causes of mode collapse in GANs:

1. catastrophic forgetting: Catastrophic forgetting refers to the phenomenon in which a model trained on a specific task forgets the knowledge it has gained while learning a new task.
2. discriminator overfitting

• Vanishing gradient

When the Generated images and Real images distribution are like each other or far from each other the gradient will be close to zero

Solution

Earth Mover’s distance:

Distance between two probability distributions. It can be interpreted as the minimum energy cost of moving and transforming a pile of dirt in the shape of one probability distribution to the shape of the other distribution.

WLOSS

BCELoss

But in here we will use the mean of the outputs so the gradient of any point won’t be zero

Wasserstein Loss

Generator Loss

Critic Loss

One matrix to calculate the loss for real input and zeros matrix to calculate the loss for fake input which is the output of generator so the BCE formular will be:

Discriminator output is between 0 and 1 Critic output not restricted to be between 0 and 1

• MinimaMax game:

Critic tries to maximize the the differences, While Generetor tries to minimize the difference

Lipschitz continuity

The differential of the function need to satisfy in every point, meaning it should be K-Lipschitz continuous. Here is known as a Lipschitz constant for function . Functions that are everywhere continuously differentiable is Lipschitz continuous, because the derivative, estimated as , has bounds. However, a Lipschitz continuous function may not be everywhere differentiable, such as .

Force the function to be 1-Lipschitz

• Weight clipping

by clipping all entries of W (output of each layer) to within some interval , we have can bound . where is the operator norm

• Gradient penalty

LSGANs

• LSGAN is proposed, in which the least squares loss function is adopted for the discriminator.
• LSGANs are able to generate higher quality images than regular GANs.beside, LSGANs perform more stable during the learning process.

Why?

1. More difficult to saturate → Better stability, Least squares loss function is flat only at one point; Sigmoid cross entropy will saturate when x is large
2. Tougher penalties → Higher Quailty

Generator Loss

Critic Loss

• a: labels for fake data; →Discriminator
• b: labels for real data; →Discriminator
• c: values that G wants D to believe; →Generator
• a, b coding scheme for the discriminator, where a and b are the labels for fake data and real data, respectively. Then the. c denotes the value that G wants D to believe for fake data.

Parameters Selection

One condition is to set b-c=1 and b-a=2 . minimizing Pearson X² divergence is equel to minimizing the above loss functions.

• For example a=-1,b=1, c=0 :

Let b = c ⇒ Generating samples as real as possible

• For example a=0, b=-1, c=-1 :

RaLSGANs

Generator Loss

Critic Loss