Generative Adversarial Networks

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Minimax Game
5. Optimal Discriminator
6. Generator Objective and Jensen-Shannon Divergence
7. Training Algorithm
8. Mode Collapse and Training Instability
9. Wasserstein GAN (WGAN)
10. Common Pitfalls
11. Research Perspective
12. Summary
13. Exercises

Learning Objectives

1. Formulate the GAN training as a two-player minimax game.
2. Derive the optimal discriminator for fixed generator.
3. Prove that at optimality, the generator minimizes $\text{JSD}(p_{\text{data}} \| p_g)$.
4. Explain mode collapse and why it occurs.
5. Derive the Wasserstein distance and the WGAN objective.

Notation

• $p_{\text{data}}(\mathbf{x})$ — true data distribution
• $p_z(\mathbf{z})$ — latent prior (e.g., $\mathcal{N}(\mathbf{0}, \mathbf{I})$)
• $G(\mathbf{z}; \theta_g)$ — generator (maps latent to data space)
• $D(\mathbf{x}; \theta_d)$ — discriminator (outputs probability of "real")
• $p_g(\mathbf{x})$ — implicit distribution induced by generator
• $\text{JSD}$ — Jensen-Shannon divergence

Core Intuition

The generator tries to produce fake samples indistinguishable from real data. The discriminator tries to tell real from fake. This adversarial game drives both networks to improve until the generator's output distribution matches the true data distribution — at which point the discriminator can do no better than random guessing.

The Minimax Game

The GAN value function:

The discriminator $D$ maximizes $V$ (correctly classifying real/fake). The generator $G$ minimizes $V$ (fooling the discriminator).

Optimal Discriminator

Theorem. For fixed $G$, the optimal discriminator is:

Proof. For fixed $G$, the objective for $D$ is:

For each $\mathbf{x}$, the integrand $a\log y + b\log(1-y)$ (with $a = p_{\text{data}}$, $b = p_g$) is maximized at:

Generator Objective and Jensen-Shannon Divergence

Theorem. With optimal discriminator $D^*$, the generator minimizes the Jensen-Shannon divergence.

Proof. Substituting $D^*$ into $V$:

Let $m = (p_{\text{data}} + p_g)/2$. Then:

The minimum is achieved when $p_g = p_{\text{data}}$, giving $\text{JSD} = 0$ and $V = -2\log 2$. $\blacksquare$

Training Algorithm

Alternating optimization:

For each training iteration:

1. Update discriminator ($k$ steps): sample real batch and fake batch; maximize $V$ w.r.t. $\theta_d$.
2. Update generator (1 step): sample latent codes; minimize $V$ w.r.t. $\theta_g$ (or equivalently, maximize $\mathbb{E}[\log D(G(\mathbf{z}))]$).

Non-saturating loss (practical modification): Instead of minimizing $\log(1-D(G(\mathbf{z})))$ (which has near-zero gradient when $D$ is confident), maximize $\log D(G(\mathbf{z}))$. This provides stronger gradients early in training.

Mode Collapse and Training Instability

Mode collapse: The generator learns to produce only a few modes of the data distribution, ignoring the rest. Mathematically, $p_g$ concentrates on a low-entropy subset.

Why it happens: The minimax game is a saddle-point problem. Gradient descent on $G$ and ascent on $D$ can oscillate rather than converge. If $D$ becomes too strong, the gradient signal to $G$ vanishes; if $G$ collapses to one mode, $D$ can only indicate "that mode is overrepresented."

Training instability indicators:

• Discriminator loss → 0 (too strong)
• Generator loss oscillates without decreasing
• Generated samples lack diversity

Wasserstein GAN (WGAN)

Replace JSD with the Wasserstein-1 (Earth Mover's) distance:

By the Kantorovich-Rubinstein duality:

where the supremum is over 1-Lipschitz functions $f$.

WGAN objective: Replace $D$ with a "critic" $f$ (no sigmoid, unconstrained output) and enforce the Lipschitz constraint:

Advantages: Wasserstein distance is continuous even when supports don't overlap (unlike JSD which saturates). Provides meaningful gradients everywhere. The critic loss correlates with sample quality.

Lipschitz enforcement: Weight clipping (original WGAN) or gradient penalty (WGAN-GP): $\lambda\mathbb{E}[(\lVert\nabla f(\hat{\mathbf{x}})\rVert - 1)^2]$.

Common Pitfalls

Pitfall 1. Training $D$ to convergence before updating $G$. If $D$ is perfect, $G$ receives no useful gradient. Balance is key.

Pitfall 2. Using batch normalization in the discriminator for WGAN. BN introduces dependencies between samples, violating the per-sample Lipschitz constraint. Use layer normalization or spectral normalization instead.

Pitfall 3. Evaluating GAN quality by discriminator loss alone. Low $D$ loss doesn't imply good generation. Use FID, IS, or visual inspection.

Research Perspective

GANs (Goodfellow et al., 2014) demonstrated that implicit generative models trained via adversarial objectives could produce sharp, realistic images. Key developments include DCGAN (architectural guidelines), Progressive GAN, StyleGAN (state-of-the-art image synthesis), and theoretical advances (WGAN, spectral normalization). Diffusion models have largely superseded GANs for image generation due to more stable training and better diversity, but GANs remain relevant for real-time generation and video.

Summary

• GANs formulate generation as a minimax game between generator and discriminator.
• The optimal discriminator recovers the likelihood ratio; the generator minimizes JSD.
• Training is a saddle-point problem — prone to oscillation and mode collapse.
• WGAN uses Wasserstein distance with a Lipschitz-constrained critic for more stable gradients.
• The global optimum is $p_g = p_{\text{data}}$ where $D^* = 1/2$ everywhere.

Exercises

Exercise 1. Verify that $D^*(\mathbf{x}) = 1/2$ for all $\mathbf{x}$ when $p_g = p_{\text{data}}$.

Exercise 2. Derive equation (5) step by step, showing the connection to JSD.

Exercise 3. Show that the non-saturating generator loss $-\log D(G(\mathbf{z}))$ minimizes $\text{KL}(p_g \| p_{\text{data}})$ under the optimal discriminator (and explain why this can cause mode collapse).

Exercise 4. Prove the Kantorovich-Rubinstein duality (equation 7) for discrete distributions.

Exercise 5. For two 1D Gaussians $p = \mathcal{N}(0,1)$ and $q = \mathcal{N}(\mu, 1)$, compute $W(p,q)$ and $\text{JSD}(p \| q)$ and show that $W$ varies smoothly with $\mu$ while JSD saturates.