Generative Adversarial Networks

Why GANs exist (motivation)

Many machine learning tasks are discriminative: given x, predict y. But in generation, we want to sample new x that look like they came from an unknown data distribution p_data(x) (images, audio, text embeddings, etc.). One classical approach is to define an explicit probabilistic model p_θ(x) and maximize likelihood. That can be hard when the data distribution is complex, multi-modal, and high-dimensional.

GANs offer a different route: instead of writing down p_θ(x) and computing likelihoods, you train a neural network to produce samples and another neural network to judge samples. The “judge” becomes a learned loss function that adapts to the generator’s current weaknesses.

The core idea (two networks)

A GAN contains two parametric functions:

• •Generator G: takes a latent noise vector z (e.g., z ∼ 𝓝(0, I)) and outputs a synthetic sample.
• •For images: G(z) ∈ ℝ^{H×W×C}
• •Think of G as a differentiable simulator.

• •Discriminator D: takes a sample x and outputs a scalar score interpreted as the probability the input is real:
• •D(x) ∈ (0, 1)
• •D is a binary classifier: real (label 1) vs fake (label 0)

Intuitively:

• •D becomes good at spotting artifacts in G’s outputs.
• •G updates to remove those artifacts.

The minimax game (zero-sum objective)

The classic GAN objective is:

V(D, G) = 𝔼_{x∼p_data}[log D(x)] + 𝔼_{z∼p(z)}[log(1 − D(G(z)))]

The game is:

max_D min_G V(D, G)

(or commonly written as min_G max_D V(D, G))

• •D wants to maximize V: assign high D(x) for real data, low D(G(z)) for generated.
• •G wants to minimize V: make D(G(z)) large so that log(1 − D(G(z))) becomes small (i.e., D thinks fakes are real).

What does “success” look like?

If training reaches an equilibrium:

• •The generated distribution p_g matches p_data.
• •D cannot tell real vs fake better than chance.

In that ideal case:

• •D(x) = 1/2 for all x in the support.
• •Samples from G are indistinguishable (statistically) from real data.

A mental model (for pacing)

Think of p_data as a complicated cloud in a high-dimensional space. G pushes forward a simple latent distribution p(z) through a neural network to produce a new distribution p_g. The discriminator learns a moving “boundary” between the two clouds. G then moves its cloud to cross that boundary.

This creates an important theme you will see repeatedly:

• •GANs are powerful because the loss is learned (D adapts).
• •GANs are hard because the target is moving (D keeps changing), and the game can become imbalanced.

By the end of this lesson, you should be able to:

• •derive the optimal discriminator for fixed G,
• •connect the minimax objective to divergence minimization,
• •explain why training is unstable and how modern variants fix it,
• •reason about practical training loops and diagnostics.

Why analyze the discriminator first?

GAN training alternates between updating D and updating G. To understand what G is really optimizing, we first ask:

If G were fixed, what discriminator D is best?

This is the “inner loop” of the minimax problem. Solving it reveals the divergence GANs implicitly minimize.

Step 1: Write the value function as an integral

Let p_g be the distribution of samples x = G(z) when z ∼ p(z). Then:

V(D, G) = 𝔼_{x∼p_data}[log D(x)] + 𝔼_{x∼p_g}[log(1 − D(x))]

Rewrite as:

V(D, G) = ∫ p_data(x) log D(x) dx + ∫ p_g(x) log(1 − D(x)) dx

Combine integrals:

V(D, G) = ∫ [ p_data(x) log D(x) + p_g(x) log(1 − D(x)) ] dx

Step 2: Optimize D pointwise

For a fixed x, define:

f_x(D(x)) = p_data(x) log D(x) + p_g(x) log(1 − D(x))

Since x is fixed, p_data(x) and p_g(x) are constants. We maximize f_x over D(x) ∈ (0, 1).

Differentiate with respect to D(x):

∂f_x/∂D = p_data(x)/D(x) − p_g(x)/(1 − D(x))

Set derivative to 0:

p_data(x)/D(x) = p_g(x)/(1 − D(x))

Solve for D*(x):

p_data(x) (1 − D(x)) = p_g(x) D(x)

p_data(x) − p_data(x) D(x) = p_g(x) D(x)

p_data(x) = (p_data(x) + p_g(x)) D*(x)

D*(x) = p_data(x) / (p_data(x) + p_g(x))

This is the optimal discriminator for a fixed generator.

Interpretation

• •If p_data(x) ≫ p_g(x), then D*(x) ≈ 1: x is likely real.
• •If p_g(x) ≫ p_data(x), then D*(x) ≈ 0: x is likely fake.
• •If p_data(x) = p_g(x), then D*(x) = 1/2.

So D is estimating a density ratio.

Step 3: Plug D* back in to see what G is minimizing

Compute:

V(D*, G) = ∫ p_data(x) log( p_data(x)/(p_data(x)+p_g(x)) ) dx

• •∫ p_g(x) log( p_g(x)/(p_data(x)+p_g(x)) ) dx

Let m(x) = (1/2)(p_data(x) + p_g(x)) be the mixture distribution. Note that:

p_data(x)/(p_data(x)+p_g(x)) = p_data(x)/(2m(x))

p_g(x)/(p_data(x)+p_g(x)) = p_g(x)/(2m(x))

Then:

V(D*, G) = ∫ p_data(x) log( p_data(x)/(2m(x)) ) dx + ∫ p_g(x) log( p_g(x)/(2m(x)) ) dx

Split out log(1/2):

V(D*, G) = ∫ p_data(x) [log(p_data(x)/m(x)) + log(1/2)] dx

• •∫ p_g(x) [log(p_g(x)/m(x)) + log(1/2)] dx

Use that ∫ p_data(x) dx = 1 and ∫ p_g(x) dx = 1:

V(D*, G) = (∫ p_data(x) log(p_data(x)/m(x)) dx) + (∫ p_g(x) log(p_g(x)/m(x)) dx) + 2 log(1/2)

Recognize KL divergences:

KL(p_data ‖ m) = ∫ p_data(x) log(p_data(x)/m(x)) dx

KL(p_g ‖ m) = ∫ p_g(x) log(p_g(x)/m(x)) dx

Thus:

V(D*, G) = KL(p_data ‖ m) + KL(p_g ‖ m) − 2 log 2

The Jensen–Shannon divergence is:

JSD(p_data ‖ p_g) = (1/2) KL(p_data ‖ m) + (1/2) KL(p_g ‖ m)

So:

V(D*, G) = 2·JSD(p_data ‖ p_g) − 2 log 2

Big conclusion

When D is optimal, minimizing V(D*, G) with respect to G is equivalent to minimizing JSD(p_data ‖ p_g). The minimum is achieved when p_g = p_data.

But there’s a practical twist (gradient issues)

The original minimax generator loss is:

L_G^minimax = 𝔼_{z∼p(z)}[log(1 − D(G(z)))]

If D becomes too good early, then D(G(z)) ≈ 0, and:

log(1 − D(G(z))) ≈ log(1) = 0

Its gradient can become very small (the generator “stalls”). A common alternative is the non-saturating generator loss:

L_G^NS = − 𝔼_{z∼p(z)}[log D(G(z))]

This has stronger gradients when D(G(z)) is small.

Summary table of common GAN losses

Why GANs are tricky (motivation)

In supervised learning, you minimize a fixed loss. In GANs, the loss depends on D, which is being updated too. So optimization is not “rolling downhill” on a static surface; it’s closer to chasing a moving target in a game.

This can create:

• •Oscillations / cycling (you improve against the current opponent, then the opponent adapts)
• •Mode collapse (G produces limited variety that fools D)
• •Vanishing gradients (D becomes too strong)
• •D overpowering G or G overpowering D (imbalance)

Understanding these issues helps you choose objectives and regularizers.

1) Mode collapse: what it is and why it happens

Symptom

G maps many latent vectors z to the same (or few) outputs:

G(z₁) ≈ G(z₂) ≈ …

So p_g covers only a subset of modes of p_data.

Why it can happen (game perspective)

D provides gradients that only punish current mistakes. If G finds a small set of outputs that D currently misclassifies as real, G can “exploit” that weakness. If D then adapts, G may hop to another exploit, producing cycling behavior.

Practical mitigations

• •Make D stronger but regularized (so it generalizes rather than memorizes)
• •Add techniques that encourage diversity (minibatch discrimination, unrolled GANs)
• •Use objectives with better geometry (e.g., Wasserstein)

2) Why Jensen–Shannon can be problematic

The JSD is well-behaved when distributions overlap, but in high dimensions, supports can be nearly disjoint early in training. Then D can perfectly separate real and fake, making gradients uninformative.

This motivates alternative distances/divergences with more useful gradients when supports don’t overlap much.

3) Wasserstein GAN (WGAN): a key conceptual fix

Why Wasserstein distance?

The Wasserstein-1 (Earth Mover) distance measures how much “mass” must move to turn one distribution into another. It can provide meaningful gradients even when supports are disjoint.

Formally:

W(p_data, p_g) = inf_{γ ∈ Π(p_data, p_g)} 𝔼_{(x,y)∼γ}[‖x − y‖]

This is hard to compute directly. WGAN uses the Kantorovich–Rubinstein duality:

W(p_data, p_g) = sup_{‖f‖_L ≤ 1} 𝔼_{x∼p_data}[f(x)] − 𝔼_{x∼p_g}[f(x)]

So instead of a discriminator that outputs probabilities, WGAN uses a critic f (often still called D) that outputs real numbers, constrained to be 1-Lipschitz.

WGAN objectives

• •Critic maximization:

max_f 𝔼_{x∼p_data}[f(x)] − 𝔼_{z∼p(z)}[f(G(z))]

• •Generator minimization:

min_G − 𝔼_{z∼p(z)}[f(G(z))]

Enforcing Lipschitzness

Original WGAN used weight clipping (crude). A widely used improvement is WGAN-GP (gradient penalty):

L_D = −(𝔼_{x∼p_data}[f(x)] − 𝔼_{z}[f(G(z))]) + λ 𝔼_{\hat{x}}[(‖∇_{\hat{x}} f(\hat{x})‖ − 1)²]

where \hat{x} are points interpolated between real and generated samples.

This penalty encourages ‖∇f‖ ≈ 1, approximating the 1-Lipschitz constraint.

4) Regularization and normalization that often matter

Even for non-WGAN GANs, regularization helps prevent D from becoming too sharp (leading to vanishing gradients).

Common tools:

5) Alternating updates (the training loop) as game solving

Why not update both simultaneously?

If you do one gradient step on both G and D, you can get rotational dynamics rather than convergence (common in games). Alternating updates approximate solving:

• •D: move toward its best response to current G
• •G: move toward its best response to current D

A typical loop:

1. 1)For k steps: update D using real x and fake G(z)
2. 2)One step: update G to improve D(G(z))

In WGAN, k is often > 1 (e.g., 5 critic steps per generator step) early in training.

Balance is a first-class design goal

If D is too weak:

• •gradients point in noisy directions; G can exploit quirks

If D is too strong:

• •gradients vanish (classic GAN) or become unhelpful due to overfitting

So “make D perfect” is not the goal; “make D a good teacher” is.

6) Diagnostics: how you know what’s going on

GANs are notoriously hard to evaluate, but you can still monitor:

• •D accuracy (if it’s ~100% always, D may be too strong; if ~50% always early, D may be too weak)
• •Loss curves (not always correlated with sample quality)
• •Visual inspection (for images)
• •Diversity checks (nearest neighbor comparisons, latent interpolations)
• •Quantitative metrics: FID, IS (imperfect but common)

A helpful habit: fix a set of latent vectors {zᵢ} and track G(zᵢ) over training. Mode collapse often shows up as many zᵢ converging to similar outputs.

Why GANs are useful in practice

GANs are most compelling when you need:

• •High perceptual quality samples (especially images)
• •Fast sampling (one forward pass through G)
• •Implicit modeling (no explicit likelihood)

Typical applications:

1) Image synthesis

• •unconditional: sample random images
• •conditional: generate a specific class or guided output

2) Image-to-image translation

• •change style, domain, or attributes
• •paired (Pix2Pix) vs unpaired (CycleGAN)

3) Super-resolution and inpainting

• •GAN loss encourages perceptual realism
• •often combined with pixel/perceptual losses

4) Data augmentation

• •generate additional training data (carefully—distribution shift can hurt)

Conditional GANs (cGANs)

Often you want control: generate x conditioned on y (label, text embedding, another image).

A simple conditional objective feeds y into both G and D:

G(z, y) → x̃

D(x, y) → probability real

Then:

max_D 𝔼_{(x,y)∼p_data}[log D(x,y)] + 𝔼_{z,y}[log(1 − D(G(z,y), y))]

Conditional setups make the mapping easier because y reduces ambiguity (less multi-modality per condition).

When GANs are not the best default

Modern diffusion models often dominate unconditional high-fidelity image generation because they are easier to train and cover modes better (at the cost of slower sampling). Autoregressive models dominate discrete sequences (text) because likelihood-based training is stable.

So a practical selection table:

Conceptual connections

GANs sit at the intersection of:

• •deep learning (neural nets as function approximators)
• •game theory (minimax equilibrium)
• •divergence minimization (implicit via D)
• •optimal transport (Wasserstein variants)

If you understand GANs deeply, you also understand a general pattern:

Learn a generator by training an adversary that provides a task-specific discrepancy signal.

That pattern reappears in domain adaptation, imitation learning (GAIL), and robust representation learning.