TinyML — Generative Adversarial Networks

From mathematical foundations to edge implementation

Social media:

👨🏽‍💻 Github: TinyML | Embedded Machine Learning Hub 👷🏾 Linkedin: Thommas Kevin | LinkedIn 🧑‍🎓Scholar: ‪Thommas Kevin Sales Flores‬ — ‪Google Académico‬ 📽 Youtube: Thommas Kevin — YouTube 👨🏻‍🏫 Research group: Conecta.ai (ufrn.br)

SUMMARY

1 — Introduction 2 — Mathematical Foundations 3 — TinyML Implementation 3.1 — Exemplo 1: Vanilla GAN — Two Moons 3.2 — Exemplo 2: WGAN-GP — Gaussian Mixture 3.3 — Exemplo 3: Conditional GAN — Three Blobs 3.4 — Exemplo 4: Hinge GAN + Spectral Norm — Sensor Data

Find for “Generative Adversarial Networks” and Give me a Star 🌟 in TinyML | Embedded Machine Learning Hub

1 — Introduction

Generative Adversarial Networks (GANs), introduced by Goodfellow et al. in 2014, are a class of deep generative models that learn to synthesize new data by framing the generation problem as a two-player game. Unlike variational autoencoders, which optimize an explicit likelihood lower bound, or normalizing flows, which require invertible architectures, a GAN places no direct constraint on the form of the generator distribution. Instead, it trains two networks in opposition: a Generator that produces synthetic samples and a Discriminator that attempts to distinguish them from real ones (Figure 01). The tension between these two networks drives the Generator to learn increasingly realistic distributions, without ever computing a likelihood.

Since their introduction, GANs have become one of the most influential frameworks in machine learning, enabling high-fidelity image synthesis, data augmentation for imbalanced datasets, domain adaptation, anomaly detection, and, increasingly, on-device synthetic data generation for embedded systems. This document develops the mathematical foundations of GANs in full, covering all loss variants implemented in this framework, and concludes with a guide to deploying a trained Generator on Arduino and ESP32 microcontrollers for TinyML applications.

Figure 01 — The GAN framework. A Generator G maps latent noise z ~ N(0,I) to synthetic samples x̃. A Discriminator D receives both real samples x ~ pdatapd​ata and fake samples x̃ ~ pGpG​, and outputs a real/fake score. Both networks are trained simultaneously: D is trained to maximize its classification accuracy, while G is trained to minimize it. The adversarial feedback loop forces G to produce increasingly realistic samples over the course of training.

1.1 — The Generative Modeling Problem

Let x∈Rd be a data point drawn from an unknown real distribution pdata​(x). The goal of generative modeling is to learn a model distribution pG​(x) that approximates pdata​(x) closely enough that samples drawn from pG​ are indistinguishable from real data (Figure 02).

Classical approaches to this problem include:

• Maximum Likelihood Estimation (MLE): Directly fit a parametric model pθ​(x) by maximizing Ex∼pdata​​[logpθ​(x)]. Requires computing a tractable likelihood, which is difficult for complex distributions in high dimensions.
• Variational Autoencoders (VAEs): Maximize a lower bound on the log-likelihood using an encoder-decoder architecture. The log-likelihood surrogate introduces a Gaussian prior that can blur generated samples.
• Normalizing Flows: Learn an invertible mapping between a simple distribution and the data distribution. Exact likelihood, but architecture is constrained to invertible functions.

GANs take a fundamentally different approach: they implicitly define pG​ through a deterministic mapping G:Rk→Rd applied to a simple noise distribution:

The distribution pGpG​ is never computed explicitly. Instead, the quality of the Generator is measured indirectly through the Discriminator, which acts as an adaptive loss function.

Figure 02 — The generative modeling problem. The goal is to learn a model distribution pG​(x) (red) that approximates the unknown real data distribution p_data(x) (blue). At the start of training (center), pGpG​ is an unstructured Gaussian. After adversarial training (right), pG​ matches the topology of the real distribution, producing samples that are statistically indistinguishable from real data.

1.2 — The Adversarial Principle

The key insight of GANs is that the distance between pdatapdata​ and pG​ can be estimated by a learned classifier rather than by an analytical formula. The Discriminator D:Rd→[0,1] is trained to output a high probability for real samples and a low probability for generated ones. Its accuracy at this task is a proxy for how dissimilar the two distributions are: if D can perfectly separate real from fake, the distributions are far apart; if D cannot do better than random chance, the distributions are identical.

This formulation is elegant for three reasons. First, it does not require computing pG​(x) explicitly. Second, the Discriminator automatically focuses on the most discriminative features of the data, providing informative gradients even in high-dimensional spaces. Third, as pG​ improves, the Discriminator’s task becomes harder, providing a natural curriculum that drives the Generator to continuously improve (Figure 03).

Figure 03 — The adversarial training curriculum. As training progresses, the Generator produces increasingly realistic samples (left column), while the Discriminator refines its decision boundary (right column). At convergence, the generated distribution matches the real distribution so closely that the Discriminator can no longer reliably distinguish them; its output approaches 0.5 everywhere, indicating that the Nash equilibrium has been reached.

1.3 — From Vanilla GANs to the Modern Landscape

The original GAN formulation of Goodfellow et al. (2014) used binary cross-entropy as the adversarial objective. While conceptually clean, this formulation suffers from training instability and mode collapse, pathologies that led to a decade of research into improved GAN training procedures (Figure 04).

The most significant advances include:

• DCGAN (Radford et al., 2015): Architectural guidelines (batch normalization, strided convolutions) that dramatically stabilize training.
• LSGAN (Mao et al., 2017): Least-squares loss that avoids gradient saturation.
• WGAN (Arjovsky et al., 2017): Wasserstein distance objective with theoretical convergence guarantees.
• WGAN-GP (Gulrajani et al., 2017): Gradient penalty to enforce the Lipschitz constraint without weight clipping.
• Spectral Normalization (Miyato et al., 2018): Per-layer normalization of Discriminator weights that enforces the Lipschitz constraint without additional terms in the loss.
• cGAN (Mirza & Osindero, 2014): Class-conditional generation by feeding labels to both networks.

This framework implements all of the above as interchangeable loss functions and layer options, allowing the user to select the appropriate variant for their dataset and deployment target.

Figure 04 — Major milestones in GAN development. Each variant addressed a specific limitation of its predecessor: DCGAN stabilized training through architectural guidelines; LSGAN and WGAN replaced BCE with losses that provide informative gradients throughout training; WGAN-GP and SN-GAN enforced Lipschitz constraints without weight clipping; and cGAN extended unconditional generation to class-conditional synthesis. This framework implements all highlighted variants as interchangeable modules.

2 — Mathematical Foundations

2.1 — The Minimax Objective

Let Gθ​:Rk→Rd be the Generator with parameters θG​ and Dϕ​:Rd→[0,1] be the Discriminator with parameters θD​. The original GAN objective is the following minimax game (Figure 05):

The Discriminator D maximizes V by assigning high logD(x) to real samples and high log(1−D(G(z))) to fake samples, i.e., low D(G(z)).

The Generator G minimizes V because it wants to maximize D(G(z)) so that the second term log(1−D(G(z))) is small.

Optimal Discriminator. For a fixed Generator GG, the optimal Discriminator is:

This is the Bayes-optimal classifier that assigns to each point the posterior probability that it came from the real distribution rather than the Generator.

Global optimum. Substituting D∗ back into V, the Generator’s optimal solution is pG​=pdata​, at which point D∗(x)=1/2 everywhere, so the Discriminator cannot do better than a coin flip. At this point, the value of the game equals −log4, and the minimax objective is equivalent to minimizing twice the Jensen-Shannon Divergence between pdatapdata​ and pG​:

Figure 05 — The minimax objective V(D, G) as a saddle-shaped surface. The Discriminator (red arrow) ascends the surface to maximize V, while the Generator (blue arrow) descends to minimize it. The Nash equilibrium (center) is a saddle point where neither player can improve unilaterally: pG​ = pdata​ and D = 0.5 everywhere. In practice, gradient descent-ascent on this surface is highly non-convex, which motivates the improved loss variants discussed in Sections 2.5–2.8.*

2.2 — The Non-Saturating Generator Loss

A critical practical issue with the minimax formulation is that early in training, when the Generator produces obviously fake samples, the Discriminator can easily assign D(G(z))≈0, making log(1−D(G(z)))≈0. The gradient of this term with respect to θG​ vanishes, providing no learning signal to the Generator, a phenomenon known as the discriminator saturation problem.

Goodfellow et al. (2014) proposed the non-saturating Generator loss as a practical fix: instead of minimizing log(1−D(G(z))), the Generator maximizes logD(G(z)):

This is equivalent to minimizing the binary cross-entropy between D(G(z)) and the label 1. The gradient is large when D(G(z))≈0 (the Generator is failing) and small when D(G(z))≈1 (the Generator is succeeding), providing informative gradients throughout training (Figure 06).

The corresponding Discriminator loss (on real and fake batches separately) is:

Figure 06 — Comparison of the saturating (original minimax) and non-saturating Generator loss. Left: in the saturating formulation, the gradient ∂LG​/∂θG​ vanishes when D(G(z)) ≈ 0, i.e., when the Generator most needs to improve. Right: the non-saturating formulation −log D(G(z)) has large gradients exactly when the Generator is failing, providing a consistently informative learning signal. All GAN variants in this framework use the non-saturating formulation for the Generator.

2.3 — Training Dynamics and the Nash Equilibrium

In the ideal theoretical analysis, the Discriminator is trained to its global optimum before each Generator update. In practice, both networks are updated by simultaneous gradient descent-ascent using mini-batches (Figure 07):

For each training step:

1. Sample x(i)∼pdata​ and z(i)∼pz​ for i=1,…,B.
2. Compute x~(i)=G(z(i)) (no gradient through G for D update).
3. Update θD​←θD​+α∇θD​​LD​ (gradient ascent on DD).
4. Sample fresh z(i)∼pz​.
5. Update θG​:θG​←θG​−α∇θG​​LG​ (gradient descent on GG).

A key design choice is how many Discriminator updates to perform per Generator update. For WGAN variants, the Critic is typically updated ncritic​=5 times per Generator step to ensure the Critic approximates the true Wasserstein distance accurately before the Generator moves. For vanilla GAN and LSGAN, ncritic​=1 is standard.

Figure 07 — The alternating gradient update loop in GAN training. In each outer iteration: (blue path) the Discriminator is updated ncriticncritic​ times on real and fake batches with G frozen; (red path) the Generator is updated once with D frozen. Separating the two updates prevents gradient interference and allows the Discriminator to provide a meaningful signal before the Generator moves. The number of Critic updates ncriticncritic​ is a key hyperparameter: larger values improve the Wasserstein distance estimate for WGAN variants at the cost of proportionally more computation.

2.4 — Failure Modes: Mode Collapse and Discriminator Saturation

GANs are notoriously difficult to train. Two failure modes dominate in practice (Figure 08):

Figure 08 — The two dominant GAN failure modes. Left: mode collapse, where the Generator produces samples concentrated in a small region of the data space, covering only one or a few modes of the real distribution. The loss curves show the Generator loss suddenly decreasing while diversity collapses. Right: Discriminator saturation, where the Discriminator becomes perfect too quickly and the Generator receives near-zero gradients. Both failure modes can be diagnosed from the loss curves and mitigated by the WGAN-GP, LSGAN, or hinge + spectral normalization objectives described in Sections 2.5–2.8.

Mode collapse. The Generator learns to produce a limited variety of outputs, sometimes only a single sample, that consistently fool the Discriminator. This occurs because the Generator can find a local strategy (mapping many different z values to the same x~) that maximizes D(G(z)) without covering the full data distribution. Diagnostically: the Generator loss decreases rapidly while the FID proxy stagnates or worsens.

Discriminator saturation. The Discriminator becomes too powerful too quickly, assigning near-zero probability to all generated samples. The Generator’s gradient vanishes (in the saturating formulation) and training stalls. Diagnostically: D loss collapses to zero early in training.

Remedies implemented in this framework:

2.5 — Least-Squares GAN (LSGAN)

Mao et al. (2017) observed that the BCE-based Discriminator assigns near-zero gradients to samples that are correctly classified but far from the decision boundary. LSGAN replaces the logarithmic loss with a quadratic (mean-squared error) objective (Figure 09):

where (a,b,c) are target values for fake, real, and generated samples respectively. The canonical choice is a=0, b=1 and c=1.

The key advantage: the quadratic loss penalizes samples that are on the correct side of the decision boundary but far from it, forcing the Generator to move fake samples toward the real data manifold rather than simply past the boundary. LSGAN is also equivalent to minimizing the Pearson **χ2 divergence* between pdata​ and pG*​, which has useful theoretical properties.

Figure 09 — Behavioral comparison of BCE and LSGAN decision boundaries. With BCE (left), correctly classified samples that lie far from the decision boundary receive near-zero gradients, so the Generator has no incentive to move these samples closer to the real data manifold. LSGAN (right) uses a quadratic penalty that grows with distance from the boundary, providing informative gradients for all generated samples regardless of their classification status. This produces smoother training curves and better sample diversity.

2.6 — Wasserstein GAN (WGAN) and the Earth Mover’s Distance

Arjovsky et al. (2017) identified the root cause of GAN instability: when pdatapdata​ and pG​ have disjoint or nearly disjoint supports (which is typical in high dimensions), the Jensen-Shannon Divergence is constant and its gradient is zero or undefined. They proposed replacing JSD with the Earth Mover’s Distance (also called Wasserstein-1 distance):

where Π is the set of all joint distributions with marginals pdatapdata​ and pG​.

The Earth Mover’s Distance has a meaningful gradient even when the two distributions do not overlap, because it measures the minimum amount of “work” (mass times distance) required to transform one distribution into the other. By the Kantorovich-Rubinstein duality theorem, W can be computed as:

where the supremum is over all 1-Lipschitz functions f. The WGAN Critic C approximates this supremum:

The Lipschitz constraint ∥f∥L​≤1 is enforced by weight clipping: after each Critic update, all parameters of C are clipped to [−c,c] for a small constant c (typically 0.01).

Figure 10 — Comparison of the Jensen-Shannon Divergence (JSD) and Earth Mover’s Distance (EMD/Wasserstein-1) for two distributions with disjoint support. When pdatapdata​ and pG​ do not overlap (left), JSD reaches its maximum value of log 2 and provides no gradient, which is the root cause of GAN instability. The EMD (right) measures the minimum transport cost to move mass from pG​ to pdatapdata​, which remains finite and meaningful even with disjoint support, providing an informative gradient signal throughout training.

2.7 — Wasserstein GAN with Gradient Penalty (WGAN-GP)

Weight clipping in WGAN introduces new problems: it forces the Critic toward simple weight configurations that use only the extreme values ±c, limiting capacity, and requires careful tuning of c. Gulrajani et al. (2017) proposed enforcing the Lipschitz constraint more directly via a gradient penalty on interpolated samples:

where x^=*εx+(1−ε)x~, ε∼Uniform(0,1) is a random interpolation between real and fake samples, and λ*=10 is the standard coefficient.

The penalty forces ∥∇x^​C(x^)∥2​=1 along straight lines between real and generated samples, which is a sufficient condition for 1-Lipschitz continuity on the interpolation paths. Crucially, WGAN-GP does not require weight clipping, and the Critic can use LayerNorm (but not BatchNorm, which introduces inter-sample dependencies that invalidate the penalty).

WGAN-GP is the most stable and widely used GAN variant for tabular and low-dimensional data, and is the recommended default in this framework (Figure 10 shows the EMD motivation).

2.8 — Hinge Loss GAN and Spectral Normalization

The Hinge GAN (Lim & Ye, 2017; Miyato et al., 2018) uses a hinge loss that enforces a margin between the real and fake scores:

The hinge loss is paired with Spectral Normalization (SN, Miyato et al. 2018), which constrains the Lipschitz constant of each Discriminator layer by dividing its weight matrix by its largest singular value σ1​(W):

SN is applied at every forward pass via power iteration, making it computationally efficient. Combined with the hinge loss, it produces one of the most stable GAN training regimes available (Figure 11).

Figure 11 — Spectral normalization (SN) constrains the Lipschitz constant of each Discriminator layer by normalizing its weight matrix W by the largest singular value σ1(W)σ1​(W), computed efficiently via power iteration at each forward pass. This ensures ∣∣W~∣∣2​=1 for every layer, bounding the overall Lipschitz constant of the network. Unlike WGAN-GP, SN does not add a gradient penalty term to the loss and is compatible with BatchNorm, making it a lightweight alternative for enforcing stability.

2.9 — Conditional GAN (cGAN)

Mirza & Osindero (2014) extended the GAN framework to class-conditional generation by providing both the Generator and Discriminator with an additional conditioning signal c (e.g., a class label):

For tabular data with integer class labels, this framework implements cGAN by:

1. Embedding the label into a dense vector e=Wemb​[c] via nn.Embedding.
2. Generator: concatenating ee to z before the first layer, and using Conditional Batch Normalization (CBN) in hidden layers, where the BN scale and shift are predicted from e.
3. Discriminator: concatenating ee to x before the first layer.

Conditional Batch Normalization for a feature vector h with label c:

where γ(c )=Wγ​e and β(c )=Wβ​e are linearly predicted from the label embedding. This allows each class to have its own effective normalization parameters, substantially increasing the Generator’s ability to produce class-specific features (Figure 12).

Figure 12 — Conditional GAN (cGAN) architecture. Both the Generator and Discriminator receive the class label c through an embedding layer that produces a dense vector e. In the Generator, e is concatenated to the latent vector z and used to modulate the batch normalization parameters at each hidden layer via Conditional Batch Normalization (CBN): γ(c ) and β(c ) are predicted from e, giving each class its own effective normalization. In the Discriminator, e is concatenated to x before the first layer. This design allows the Generator to produce class-specific samples that are evaluated by a class-aware Discriminator.

2.10 — Regularization: Feature Matching, R1 Penalty, Mode Seeking

Beyond the main loss objectives, three regularization techniques are implemented in vi.py:

Feature Matching (Salimans et al., 2016): Instead of maximizing D(G(z)) directly, the Generator is trained to match intermediate Discriminator activation statistics between real and fake batches:

where fℓ​(⋅) denotes the ℓ-th layer activations of the Discriminator. This stabilizes training and reduces mode collapse.

R1 Gradient Penalty (Mescheder et al., 2018): Penalizes the gradient of the Discriminator on real data only:

Unlike WGAN-GP, R1 does not require interpolated samples and converges for a broader class of GAN objectives.

Mode Seeking (Mao et al., 2019): Discourages the Generator from mapping different latent vectors to the same output:

Adding LMS​ to the Generator loss directly penalizes mode collapse by requiring the Generator’s output to be diverse relative to the diversity of its inputs.

3 — TinyML Implementation

With this example you can implement the machine learning algorithm in ESP32, Arduino, Arduino Portenta H7 with Vision Shield, Raspberry and other different microcontrollers or IoT devices.

3.1 — Clone repository (🌟Give me a Star)

Find for “Generative Adversarial Networks” and Give me a Star 🌟 in TinyML | Embedded Machine Learning Hub

3.2 — Install the libraries listed in the requirements.txt file

!pip install -r requirements.txt

3.3 — Importing Libraries

import sys, os
sys.path.append('37_GAN')   # adjust if running from a different directory

import torch
import torch.optim as optim
import numpy as np
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import make_moons, make_blobs
from sklearn.preprocessing import StandardScaler

from model  import Generator, Discriminator, ConditionalGAN
from layers import get_activation
from losses import compute_discriminator_loss, compute_generator_loss, LOSS_TYPES
from utils  import (
    export_to_json,
    GANTrainer,
    plot_training_history,
    plot_generated_samples,
    plot_latent_interpolation,
    plot_loss_landscape,
    evaluate_fid_proxy,
)
from cpp_generator import generate_ino, add_bn_stats_to_json

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Device:', DEVICE)
print('Available loss types:', LOSS_TYPES)

os.makedirs('json_model',   exist_ok=True)
os.makedirs('arduino_code', exist_ok=True)

3.4 — Example 1: Vanilla GAN on Two Moons

The classic make_moons dataset (2-D) demonstrates the basic GAN training loop. The Generator learns to produce samples that match the two-crescent distribution from 2-D Gaussian noise.

def train_vanilla_moons():
    print('=== Example 1 — Vanilla GAN: Two Moons ===')
    torch.manual_seed(42); np.random.seed(42)

    # ---- Data ----
    X, _ = make_moons(n_samples=4000, noise=0.05, random_state=42)
    scaler = StandardScaler()
    X = scaler.fit_transform(X).astype(np.float32)
    dataset = TensorDataset(torch.FloatTensor(X))
    loader  = DataLoader(dataset, batch_size=128, shuffle=True, drop_last=True)

    LATENT_DIM = 2

    # ---- Architecture ----
    G = Generator(
        latent_dim=LATENT_DIM,
        generator_layers=[
            {'out_features': 128, 'activation': 'relu',   'use_bn': True},
            {'out_features': 256, 'activation': 'relu',   'use_bn': True},
            {'out_features': 2,   'activation': 'tanh',   'use_bn': False},
        ],
    )
    D = Discriminator(
        input_dim=2,
        discriminator_layers=[
            {'out_features': 256, 'activation': 'leaky_relu', 'dropout': 0.3},
            {'out_features': 128, 'activation': 'leaky_relu', 'dropout': 0.3},
            {'out_features': 1,   'activation': 'linear'},
        ],
    )

    g_opt = optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
    d_opt = optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

    trainer = GANTrainer(
        G, D, g_opt, d_opt,
        loss_type='vanilla',
        device=DEVICE,
        label_smoothing=0.1,
    )

    # ---- Train ----
    d_losses, g_losses = trainer.train(loader, epochs=100, print_every=50)

    # ---- Visualize ----
    plot_training_history(d_losses, g_losses, loss_type='vanilla')
    plot_generated_samples(G, X, n_samples=2000, device=DEVICE,
                           title='Vanilla GAN — Two Moons')
    plot_latent_interpolation(G, n_steps=12, n_pairs=4, device=DEVICE,
                              title='Latent Space Interpolation — Two Moons')

    fid = evaluate_fid_proxy(G, X, device=DEVICE)
    print(f'FID proxy: {fid:.4f}')

    # ---- Export ----
    G.eval()
    export_to_json(G, 'json_model/vanilla_moons_gen.json')
    add_bn_stats_to_json(G, 'json_model/vanilla_moons_gen.json')
    generate_ino('json_model/vanilla_moons_gen.json',
                 'arduino_code/vanilla_moons_ino', board='esp32')
    return G, D

G1, D1 = train_vanilla_moons()

3.4.1 — Deploy in Microcontroller

/*
 * GAN Generator -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Latent dim  : 2
 * Output dim  : 2
 * Conditional : False
 *
 * Input z (first 8 shown): [1.764052, 0.400157]
 * Expected output (first 8): [-0.997706, 0.999916]
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed values match to at least 4 decimal places.
 */

#include "GANModel.h"

GANModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);

    const int LATENT_DIM = 2;
    const int OUTPUT_DIM = 2;

    // Verification latent vector (auto-generated by Python exporter)
    float z[LATENT_DIM] = {
        1.76405239f, 0.40015721f
    };

    float out[OUTPUT_DIM];
    model.generate(z, out);

    Serial.println("Generated output (first 8 dims):");
    int n_print = OUTPUT_DIM < 8 ? OUTPUT_DIM : 8;
    for (int i = 0; i < n_print; i++) {
        Serial.print("  out["); Serial.print(i);
        Serial.print("] = "); Serial.println(out[i], 6);
    }
}

void loop() {
    // Nothing to do here
}

3.5 — Example 2: WGAN-GP on Gaussian Mixture

Wasserstein GAN with gradient penalty on a 2-D mixture of 5 Gaussians. WGAN-GP provides more stable training and better mode coverage than vanilla GAN. The Critic (not Discriminator) is updated 5 times per Generator step.

def train_wgan_gp_gmm():
    print('=== Example 2 — WGAN-GP: Gaussian Mixture ===')
    torch.manual_seed(42); np.random.seed(42)

    # ---- Data: 5-component Gaussian mixture ----
    centers = np.array([[0,0],[2,2],[-2,2],[2,-2],[-2,-2]], dtype=np.float32)
    parts = [np.random.randn(800, 2).astype(np.float32) * 0.3 + c for c in centers]
    X = np.concatenate(parts, axis=0)
    np.random.shuffle(X)

    dataset = TensorDataset(torch.FloatTensor(X))
    loader  = DataLoader(dataset, batch_size=128, shuffle=True, drop_last=True)

    LATENT_DIM = 8

    # ---- Architecture ----
    G = Generator(
        latent_dim=LATENT_DIM,
        generator_layers=[
            {'out_features': 256, 'activation': 'relu',   'use_bn': True},
            {'out_features': 256, 'activation': 'relu',   'use_bn': True},
            {'out_features': 2,   'activation': 'linear', 'use_bn': False},
        ],
    )
    D = Discriminator(
        input_dim=2,
        discriminator_layers=[
            {'out_features': 256, 'activation': 'leaky_relu', 'use_ln': True},
            {'out_features': 256, 'activation': 'leaky_relu', 'use_ln': True},
            {'out_features': 1,   'activation': 'linear'},
        ],
    )

    g_opt = optim.Adam(G.parameters(), lr=1e-4, betas=(0.0, 0.9))
    d_opt = optim.Adam(D.parameters(), lr=1e-4, betas=(0.0, 0.9))

    trainer = GANTrainer(
        G, D, g_opt, d_opt,
        loss_type='wgan_gp',
        device=DEVICE,
        n_critic=5,
        lambda_gp=10.0,
    )

    # ---- Train ----
    d_losses, g_losses = trainer.train(loader, epochs=100, print_every=75)

    # ---- Visualize ----
    plot_training_history(d_losses, g_losses, loss_type='wgan_gp')
    plot_generated_samples(G, X, n_samples=2000, device=DEVICE,
                           title='WGAN-GP — 5-Component Gaussian Mixture')
    plot_loss_landscape(d_losses, g_losses,
                        title='WGAN-GP Loss Landscape — Gaussian Mixture')

    fid = evaluate_fid_proxy(G, X, device=DEVICE)
    print(f'FID proxy: {fid:.4f}')

    # ---- Export ----
    G.eval()
    export_to_json(G, 'json_model/wgan_gp_gmm_gen.json')
    add_bn_stats_to_json(G, 'json_model/wgan_gp_gmm_gen.json')
    generate_ino('json_model/wgan_gp_gmm_gen.json',
                 'arduino_code/wgan_gp_gmm_ino', board='esp32')
    return G, D

G2, D2 = train_wgan_gp_gmm()

3.5.1 — Deploy in Microcontroller

/*
 * GAN Generator -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Latent dim  : 8
 * Output dim  : 2
 * Conditional : False
 *
 * Input z (first 8 shown): [1.764052, 0.400157, 0.978738, 2.240893, 1.867558, -0.977278, 0.950088, -0.151357]
 * Expected output (first 8): [-0.028695, -0.308894]
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed values match to at least 4 decimal places.
 */

#include "GANModel.h"

GANModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);

    const int LATENT_DIM = 8;
    const int OUTPUT_DIM = 2;

    // Verification latent vector (auto-generated by Python exporter)
    float z[LATENT_DIM] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f, 1.86755800f, -0.97727787f, 0.95008844f, -0.15135720f
    };

    float out[OUTPUT_DIM];
    model.generate(z, out);

    Serial.println("Generated output (first 8 dims):");
    int n_print = OUTPUT_DIM < 8 ? OUTPUT_DIM : 8;
    for (int i = 0; i < n_print; i++) {
        Serial.print("  out["); Serial.print(i);
        Serial.print("] = "); Serial.println(out[i], 6);
    }
}

void loop() {
    // Nothing to do here
}

3.6 — Example 3: Conditional GAN (cGAN) on Blobs

A class-conditional GAN generates samples conditioned on an integer class label. Both the Generator and Discriminator receive the label as an additional embedding.

def train_cgan_blobs():
    print('=== Example 3 — Conditional GAN (cGAN): Three Blobs ===')
    torch.manual_seed(42); np.random.seed(42)

    NUM_CLASSES = 3
    LATENT_DIM  = 4

    # ---- Data ----
    X, y = make_blobs(n_samples=3000, centers=3, cluster_std=0.8, random_state=42)
    scaler = StandardScaler()
    X = scaler.fit_transform(X).astype(np.float32)
    y = y.astype(np.int64)

    dataset = TensorDataset(torch.FloatTensor(X), torch.LongTensor(y))
    loader  = DataLoader(dataset, batch_size=128, shuffle=True, drop_last=True)

    # ---- Architecture ----
    G = Generator(
        latent_dim=LATENT_DIM,
        generator_layers=[
            {'out_features': 128, 'activation': 'relu',   'use_bn': True,
             'num_classes': NUM_CLASSES, 'embed_dim': 32},
            {'out_features': 128, 'activation': 'relu',   'use_bn': True,
             'num_classes': NUM_CLASSES, 'embed_dim': 32},
            {'out_features': 2,   'activation': 'linear', 'use_bn': False},
        ],
        num_classes=NUM_CLASSES,
        embed_dim=32,
    )
    D = Discriminator(
        input_dim=2,
        discriminator_layers=[
            {'out_features': 128, 'activation': 'leaky_relu', 'dropout': 0.2},
            {'out_features': 64,  'activation': 'leaky_relu', 'dropout': 0.2},
            {'out_features': 1,   'activation': 'linear'},
        ],
        num_classes=NUM_CLASSES,
        embed_dim=32,
    )

    g_opt = optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
    d_opt = optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

    trainer = GANTrainer(
        G, D, g_opt, d_opt,
        loss_type='lsgan',
        device=DEVICE,
    )

    # ---- Train ----
    d_losses, g_losses = trainer.train(loader, epochs=250, print_every=50)

    # ---- Visualize per-class generation ----
    plot_training_history(d_losses, g_losses, loss_type='lsgan')

    for cls in range(NUM_CLASSES):
        lbl = torch.full((500,), cls, dtype=torch.long, device=DEVICE)
        plot_generated_samples(
            G, X[y == cls], n_samples=500, device=DEVICE,
            labels=lbl, num_classes=NUM_CLASSES,
            title=f'cGAN — Class {cls} Samples',
        )

    fid = evaluate_fid_proxy(G, X, device=DEVICE)
    print(f'FID proxy (unconditional): {fid:.4f}')

    # ---- Export (unconditional path only — cGAN BN not supported in C) ----
    print('Note: cGAN uses ConditionalBatchNorm which is not exported to C.')
    print('For embedded deployment, retrain without ConditionalBatchNorm.')
    return G, D

G3, D3 = train_cgan_blobs()

3.6.1 — Deploy in Microcontroller

/*
 * GAN Generator -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Latent dim  : 4
 * Output dim  : 2
 * Conditional : True
 * Classes     : 3  (using label=0 for verification)
 *
 * Input z (first 8 shown): [1.764052, 0.400157, 0.978738, 2.240893]
 * Expected output (first 8): [0.745394, 0.139411]
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed values match to at least 4 decimal places.
 */

#include "GANModel.h"

GANModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);

    const int LATENT_DIM = 4;
    const int OUTPUT_DIM = 2;

    // Verification latent vector (auto-generated by Python exporter)
    float z[LATENT_DIM] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f
    };

    float out[OUTPUT_DIM];
    model.generate(z, 0, out);

    Serial.println("Generated output (first 8 dims):");
    int n_print = OUTPUT_DIM < 8 ? OUTPUT_DIM : 8;
    for (int i = 0; i < n_print; i++) {
        Serial.print("  out["); Serial.print(i);
        Serial.print("] = "); Serial.println(out[i], 6);
    }
}

void loop() {
    // Nothing to do here
}

3.7 — Example 4: Hinge GAN with Spectral Normalization on Sensor Data

Hinge loss combined with spectral normalization in the Discriminator, applied to a 4-D synthetic sensor dataset (e.g., accelerometer readings). The trained Generator is exported to Arduino C++ for on-device data augmentation.

def train_hinge_sensor():
    print('=== Example 4 — Hinge GAN + Spectral Norm: 4-D Sensor Data ===')
    torch.manual_seed(42); np.random.seed(42)

    # ---- Synthetic 4-D sensor data ----
    N  = 4000
    t  = np.random.uniform(0, 2 * np.pi, N).astype(np.float32)
    X  = np.stack([
        np.sin(t) + 0.05 * np.random.randn(N).astype(np.float32),
        np.cos(t) + 0.05 * np.random.randn(N).astype(np.float32),
        np.sin(2 * t) + 0.05 * np.random.randn(N).astype(np.float32),
        np.cos(2 * t) + 0.05 * np.random.randn(N).astype(np.float32),
    ], axis=1).astype(np.float32)

    dataset = TensorDataset(torch.FloatTensor(X))
    loader  = DataLoader(dataset, batch_size=128, shuffle=True, drop_last=True)

    LATENT_DIM = 16

    # ---- Architecture ----
    G = Generator(
        latent_dim=LATENT_DIM,
        generator_layers=[
            {'out_features': 128, 'activation': 'relu',   'use_bn': True},
            {'out_features': 256, 'activation': 'relu',   'use_bn': True},
            {'out_features': 4,   'activation': 'tanh',   'use_bn': False},
        ],
    )
    D = Discriminator(
        input_dim=4,
        discriminator_layers=[
            {'out_features': 256, 'activation': 'leaky_relu',
             'spectral_norm': True},
            {'out_features': 128, 'activation': 'leaky_relu',
             'spectral_norm': True},
            {'out_features': 1,   'activation': 'linear',
             'spectral_norm': True},
        ],
    )

    g_opt = optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
    d_opt = optim.Adam(D.parameters(), lr=4e-4, betas=(0.5, 0.999))

    trainer = GANTrainer(
        G, D, g_opt, d_opt,
        loss_type='hinge',
        device=DEVICE,
        n_critic=2,
    )

    # ---- Train ----
    d_losses, g_losses = trainer.train(loader, epochs=100, print_every=75)

    # ---- Visualize (first 2 dims) ----
    plot_training_history(d_losses, g_losses, loss_type='hinge')
    plot_generated_samples(G, X, n_samples=2000, device=DEVICE,
                           title='Hinge GAN — 4-D Sensor Data (dims 1 & 2)')

    fid = evaluate_fid_proxy(G, X, device=DEVICE)
    print(f'FID proxy: {fid:.4f}')

    # ---- Export ----
    G.eval()
    export_to_json(G, 'json_model/hinge_sensor_gen.json')
    add_bn_stats_to_json(G, 'json_model/hinge_sensor_gen.json')
    generate_ino('json_model/hinge_sensor_gen.json',
                 'arduino_code/hinge_sensor_ino', board='esp32')
    return G, D

G4, D4 = train_hinge_sensor()

3.7.1 — Deploy in Microcontroller

/*
 * GAN Generator -- Arduino verification sketch
 * Generated automatically -- do not edit the weights.
 *
 * VERIFICATION GUIDE
 * -------------------
 * Latent dim  : 16
 * Output dim  : 4
 * Conditional : False
 *
 * Input z (first 8 shown): [1.764052, 0.400157, 0.978738, 2.240893, 1.867558, -0.977278, 0.950088, -0.151357]
 * Expected output (first 8): [-0.483521, -0.286410, 0.603639, -0.506033]
 *
 * Upload this sketch, open Serial Monitor at 115200 baud,
 * and confirm the printed values match to at least 4 decimal places.
 */

#include "GANModel.h"

GANModel model;

void setup() {
    Serial.begin(115200);
    while (!Serial);

    const int LATENT_DIM = 16;
    const int OUTPUT_DIM = 4;

    // Verification latent vector (auto-generated by Python exporter)
    float z[LATENT_DIM] = {
        1.76405239f, 0.40015721f, 0.97873801f, 2.24089313f, 1.86755800f, -0.97727787f, 0.95008844f, -0.15135720f, -0.10321885f, 0.41059852f, 0.14404356f, 1.45427346f, 0.76103771f, 0.12167501f, 0.44386324f, 0.33367434f
    };

    float out[OUTPUT_DIM];
    model.generate(z, out);

    Serial.println("Generated output (first 8 dims):");
    int n_print = OUTPUT_DIM < 8 ? OUTPUT_DIM : 8;
    for (int i = 0; i < n_print; i++) {
        Serial.print("  out["); Serial.print(i);
        Serial.print("] = "); Serial.println(out[i], 6);
    }
}

void loop() {
    // Nothing to do here
}

References

[1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems (NeurIPS), 27.

[2] Mirza, M., & Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv:1411.1784.

[3] Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (DCGAN). arXiv:1511.06434.

[4] Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017). Least Squares Generative Adversarial Networks. ICCV 2017.

[5] Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. Proceedings of ICML 2017.

[6] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved Training of Wasserstein GANs. NeurIPS 2017.

[7] Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). Spectral Normalization for Generative Adversarial Networks. ICLR 2018.

[8] Lim, J. H., & Ye, J. C. (2017). Geometric GAN. arXiv:1705.02894.

[9] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved Techniques for Training GANs. NeurIPS 2016.

[10] Mescheder, L., Geiger, A., & Nowozin, S. (2018). Which Training Methods for GANs Do Actually Converge? ICML 2018.

[11] Mao, Q., Lee, H.-Y., Tseng, H.-Y., Ma, S., & Yang, M.-H. (2019). Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis. CVPR 2019.

[12] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. NeurIPS 2016.

[13] Larsen, A. B. L., Sønderby, S. K., Larochelle, H., & Winther, O. (2016). Autoencoding beyond Pixels Using a Learned Similarity Metric (VAE-GAN). ICML 2016.

[14] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID). NeurIPS 2017.

[15] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.