From Noise to Image: The Math Behind Generative Adversarial Networks

Math behind GANs

Hi folks! I’m back with a fascinating concept that’s been gaining a lot of attention across various domains — Generative Adversarial Networks (GANs). These powerful models are not only transforming how machines generate data but are also reshaping the way we approach research problems. Recently, I had the opportunity to explore one of the GAN variants in depth, and I was amazed , not just by how it works, but by its potential to push the boundaries of creativity and data generation in AI.

Before diving into GANs, it’s important to understand the shift they brought to the machine learning landscape. Before GANs came into picture i.e prior 2014, most machine learning models were discriminative — they focused on learning the boundary between different classes and were widely used for classification and regression tasks. These models excel at answering questions like “What is this?” but often fall short when asked “Can you create something like this?” That’s where generative models come into picture. Unlike discriminative models, generative models learn the underlying distribution of data, enabling them to generate entirely new, realistic-looking samples. The introduction of GANs by Ian Goodfellow in June 2014, marked the true beginning of the creative era in deep learning.

GANs are useful when we want to build a ML model but we have limited amount of data to work with. Therefore, GANs have the ability to generate synthetic realistic data based on prior data. One such fascinating use of GANs is that, they have been used to create hyper-realistic images of people that don’t actually exist !! Beyond this, GANs have been applied in generative design that enable you create cool 3d furniture for your home. Companies like Adobe are using GANs to build next-generation Photoshop tools, Google uses them for text generation, IBM uses them for data augmentation, and platforms like Snapchat have been using GANs to create image filters for quite a while now.

In this blog, I’ll walk through the math behind how GANs work, because understanding the why behind GANs is quite important. A solid grasp of the underlying mathematics helps break free from the black-box view of deep learning and opens up opportunities to improve or even design new generative models. I also highly recommend reading the original 2014 paper by Ian Goodfellow that introduced GANs to the world here.

Adversarial Training of GANs

A Generative Adversarial Network (GAN) is a deep learning framework consisting of two competing neural networks: the Generator and the Discriminator, trained in a game-theoretic manner. Generator uses the noise from a normal or Gaussian distribution as input and generates samples. The second model which is known as discriminator receives samples from the generator and the training data. According to game theory, the generator is trained to produce an image that looks like a real image, whereas the discriminator is learning to discriminate perfectly from generated data to actual data.

The training objective of a Generative Adversarial Network (GAN) is formulated as a two-player minimax game, as shown in below equation. The discriminator D aims to maximize the probability of correctly distinguishing real samples** *x from fake samples generated by the generator G(z), where z is sampled from a latent noise distribution. Conversely, the generator *G tries to minimize this objective by producing realistic samples that can fool the discriminator into classifying them as real. This adversarial optimization ensures that G learns to approximate the true data distribution while D improves its ability to detect generated samples.

Value function

Looking at this equation, you’ll notice that it closely resembles the binary cross-entropy loss function, which is given by:

Binary cross entropy loss

• Y represents the true label for the i-th sample
• Y-hat represents the prediction from the discriminator
• n is the number of samples in the dataset or batch

The value function is derived from the binary cross-entropy loss. The binary cross-entropy objective function provides a measure of how far we are from achieving our goal of correctly classifying real and fake samples.

Derivation of BCE Loss

The left part of the BCE loss indicated in green only matters when the ground truth of the data sample is real or 1. Meanwhile only the right part of the loss function matters when the ground truth of the data sample is fake or 0. Looking at the image below when y = 1 , the loss function is close to 0 when the prediction is close to 1 and infinity when the prediction is close to 0 .When y = 0 , the loss function is close to 0 when the prediction is close to 0 and infinity when the prediction is close to 1.

Prediction and Loss for BCE loss function

Adding both parts of the binary cross-entropy loss ; the case when the ground truth is 1 (real) and when it is 0 (fake) , gives us the value function used in Generative Adversarial Networks (GANs). Since this value function must be computed over many data points, we use expectations (denoted by E) to represent the average loss over the data distribution.

In simple terms, the discriminator aims to maximize this loss function by correctly classifying real as real and fake as fake and the generator aims to minimize it by fooling the discriminator into thinking the fake data is real.

1. Discriminator D Maximizes V(D,G): The goal is to distinguish between real and fake data. Makes D(x)≈1 for real samples and D(G(z))≈0 for fake samples.
2. Generator G Minimizes V(D,G): The goal is to fool the discriminator. Wants D(G(z))≈1 , so that generated samples look real.

• Note: G does not affect D(x), the term for real data.

Training Generative Adversarial Networks

GAN training Algorithm

This image is straight from the original 2014 GAN paper, the one that started it all! It shows the core algorithm behind how GANs are trained. Don’t worry if it looks a bit math-heavy , I’m going to break it down in plain English so it’s easy to follow. Take a moment to glance through it, and if you’re curious, I highly recommend checking out the full paper too — it’s a classic in AI research!

How GANs are trained?

Training the Discriminator:

• First, we generate some random noise and pass it through the generator to produce fake data.
• We then take a batch of real data from the actual dataset.
• Both the real and fake data are fed into the discriminator, which tries to tell them apart.
• The discriminator’s weights are updated (using gradient ascent) so it gets better at distinguishing real from fake.

Training the Generator:

• Next, we freeze the discriminator and focus on improving the generator.
• New noise is fed to the generator to create fresh fake samples.
• These are passed to the discriminator, but this time, we want the discriminator to get fooled!
• The generator’s weights are updated (using gradient descent) to produce data that looks more realistic and can trick the discriminator.

This process repeats over and over, making both the generator and discriminator better with each cycle.

Architecture of Generator and Discriminator

• The generator is a simple feedforward neural network with one hidden layer, a ReLU activation function, and a tanh activation function at the output layer to squash the outputs to values between -1 and 1, representing pixel intensity values of the images.
• The discriminator is a simple feedforward neural network with just one hidden layer . The hidden layer used leaky RELU activation functions while the output layer had just one neuron with a sigmoid activation to squash the outputs to probabilities of being fake (0) or real (1).

At the beginning of training, neither the discriminator nor the generator has any real idea of what they’re doing. The discriminator’s weights and biases are updated, while the generator’s parameters are frozen. Then, the generator’s weights and biases are updated while the discriminator’s parameters are frozen. This loop keeps repeating during adversarial training, and over time, the generator’s data distribution matches that of the training data, meanwhile the discriminator becomes a straight blue line that outputs 0.5 for all data samples. At this point, the generator has learned the data distribution of the training set and can generate data samples that look similar to the real ones.

Thanks for sticking around till the end! 🙌 GANs might seem tricky at first, but with a bit of patience, the logic behind their training and architecture becomes quite intuitive. Understanding how the generator and discriminator play this clever game helps unlock the magic behind AI-generated images and beyond. If you’re just getting started with GANs, don’t worry , take your time experimenting and exploring. The best way to learn is by building and tweaking things yourself.

Happy learning , and keep creating!

References

1. The math of GANs Explained
2. Understanding GAN