KL Divergence

Why we need a special “distance” for distributions

When you compare two vectors, Euclidean distance is natural. For probability distributions, Euclidean distance often hides what actually matters for learning and decision-making:

• •If your model assigns tiny probability to events that happen under the true distribution, your log-loss explodes.
• •If your model assigns probability mass to impossible events, you waste capacity—but you might not be punished as severely.

So we want a measure that is prediction-aware: it should tell us how costly it is to pretend the world follows $Q$ when it actually follows $P$.

KL divergence (Kullback–Leibler divergence), also called relative entropy, does exactly that.

Definition (discrete)

Let $P$ and $Q$ be distributions over the same discrete set \(\mathcal{X}\). The KL divergence from $Q$ to $P$ is

A few immediate notes:

• •The expectation is with respect to $P$ (the “true” distribution in the common interpretation).
• •The log base sets units: base 2 → bits, base $e$ → nats.
• •If there exists an $x$ with $P(x)>0$ but $Q(x)=0$, then $\log \frac{P(x)}{Q(x)}=\infty$ and KL becomes infinite.

Definition (continuous)

For densities $p(x)$ and $q(x)$ (with respect to the same base measure),

The same “support mismatch” rule applies: if $p(x)>0$ on a region where $q(x)=0$, the KL is infinite.

The most important intuition: a log ratio averaged under reality

KL is an average of a log ratio:

• •If $Q$ assigns lower probability than $P$ to likely events, then $P(x)/Q(x)$ is large → log is positive → KL grows.
• •If $Q$ assigns higher probability than $P$ to likely events, then the log ratio can be negative for those $x$.

So why isn’t KL sometimes negative overall? Because of a deep inequality (Jensen / Gibbs) that forces the expectation to be ≥ 0.

Guided visualization on the interactive canvas (do this now)

Canvas A: Bernoulli slider

1. 1)Choose a Bernoulli true distribution: $P=\mathrm{Bern}(p)$.
2. 2)Choose a model distribution: $Q=\mathrm{Bern}(q)$.
3. 3)Slide $p$ and $q$ independently.

Prompted observations:

• •Hold $p=0.2$ fixed. Move $q$ toward 0. Watch $D_{\mathrm{KL}}(P\|\|Q)$ rise sharply.
• •Now move $q$ toward 1. It also rises, but the blow-up is much more dramatic when $Q$ places near-zero probability on outcomes that happen under $P$.
• •Set $q=0$ exactly while $p>0$. The KL becomes infinite.

This is the first “feel” for what KL cares about: being wrong in the direction of underestimating true events is catastrophic in log-loss terms.

Static anchor diagram: support mismatch

Imagine two distributions over the same line:

• •$p(x)$ is concentrated on an interval, say $[0,1]$.
• •$q(x)$ is concentrated on $[1,2]$.

They barely overlap. On $[0,1]$, $p(x)>0$ but $q(x)=0$, so $D_{\mathrm{KL}}(P\|\|Q)=\infty$.

If you overlay them, the key highlighted region is where $p(x)$ has mass but $q(x)$ is zero. KL treats that as an absolute failure, because it corresponds to assigning impossible probability to events that occur.

What KL is not

• •It is not symmetric: generally $D_{\mathrm{KL}}(P\|\|Q)\neq D_{\mathrm{KL}}(Q\|\|P)$.
• •It is not a metric: it doesn’t satisfy the triangle inequality.
• •It is not simply “area between curves.” It is an expectation of log ratios, weighted by $P$.

Keep those distinctions in mind; they will matter when we interpret mode-seeking vs mode-covering behavior later.

Why before how: prediction is about log-loss

In probabilistic modeling, a standard way to score a model $Q$ on data drawn from $P$ is negative log-likelihood (log-loss). For an outcome $x$, the log-loss under model $Q$ is

The expected log-loss under the true distribution $P$ is

This quantity shows up everywhere:

• •maximum likelihood estimation (MLE)
• •cross-entropy loss in classification
• •coding and compression

So the natural question becomes:

How much worse is it to use $Q$ than to use the true $P$, on average?

Derivation: KL is the excess expected log-loss

Start from KL:

Expand the log ratio:

Plug in:

Rewrite each term:

• •Entropy: $H(P) = -\sum_x P(x)\log P(x)$
• •Cross-entropy: $H(P,Q) = -\sum_x P(x)\log Q(x)$

So

Interpretation:

• •$H(P)$ is the best achievable expected log-loss if you predict with the true $P$.
• •$H(P,Q)$ is the expected log-loss if you predict with $Q$ while data comes from $P$.
• •Their difference is the penalty: the average extra log-loss per sample.

In base 2 logs, it is literally “extra bits per symbol.”

Nonnegativity: why you can’t beat the truth on average

The statement $D_{\mathrm{KL}}(P\|\|Q)\ge 0$ formalizes: on average, you cannot do better (in expected log-loss) than predicting with the true distribution.

A clean proof uses Jensen’s inequality on the concave log function.

Let’s show the common Gibbs inequality form.

We want to show:

Equivalently,

Now use Jensen (log is concave):

Therefore,

with equality iff $P(x)=Q(x)$ for all $x$ where $P(x)>0$.

Guided visualization: “extra log-loss” as a bar chart

Canvas B: per-outcome contribution plot

For a small discrete space (say 5 outcomes), plot bars of

Prompts:

1. 1)Make one outcome very likely under $P$ (e.g., $P(x\_1)=0.6$), but set $Q(x\_1)=0.2$.

• •Watch that bar dominate KL.

2. 2)Make $Q$ larger than $P$ on a rare outcome.

• •You may see a negative contribution there, but it usually doesn’t compensate much because it’s weighted by small $P(x)$.

This visually explains why KL focuses on being accurate where $P$ puts mass.

Connection to MLE (prerequisite bridge)

Suppose you have data $x₁,\dots,x_n \sim P$ i.i.d., and a parametric model $Q_\theta$.

The average negative log-likelihood is

As $n\to\infty$, this converges to

Since

minimizing expected NLL over $\theta$ is the same as minimizing $D_{\mathrm{KL}}(P\|\|Q_\theta)$ (because $H(P)$ does not depend on $\theta$).

So MLE is “KL minimization from data distribution to model distribution.”

That phrasing becomes extremely useful later (cross-entropy, VAEs, variational inference).

Why asymmetry matters

Because KL is expectation-weighted under the first argument, swapping arguments changes what errors are emphasized.

• •Forward KL: $D_{\mathrm{KL}}(P\|\|Q)$ weights errors by $P$ (truth-weighted).
• •Reverse KL: $D_{\mathrm{KL}}(Q\|\|P)$ weights errors by $Q$ (model-weighted).

This is not just a mathematical curiosity: it determines whether an approximation spreads out to “cover” all modes or collapses onto one.

Support and the “infinite wall”

Recall:

• •If $P(x)>0$ and $Q(x)=0$ anywhere, then $D_{\mathrm{KL}}(P\|\|Q)=\infty$.
• •If $Q(x)>0$ and $P(x)=0$ somewhere, then $D_{\mathrm{KL}}(Q\|\|P)=\infty$.

So each direction imposes a different feasibility constraint:

Guided visualization: toggle forward vs reverse KL on a bimodal target

Canvas C: mixture of Gaussians vs single Gaussian approximation

1. 1)Let $P$ be a bimodal distribution (two separated Gaussians).
2. 2)Let $Q$ be a single Gaussian with adjustable mean/variance.
3. 3)Add a toggle: compute either $D_{\mathrm{KL}}(P\|\|Q)$ (forward) or $D_{\mathrm{KL}}(Q\|\|P)$ (reverse).

Prompts:

• •In forward KL mode, increase Q’s variance. You’ll often see the best fit broaden to cover both modes. Missing one mode is costly because $P$ puts mass there.
• •In reverse KL mode, the best fit often collapses onto one mode (mode-seeking). Why? Because the expectation is under $Q$: if $Q$ puts almost no mass near the other mode, it doesn’t “feel” that region.

This is a central intuition used in variational inference: minimizing reverse KL (common in VI) tends to be mode-seeking.

A simple discrete example of asymmetry

Let $\mathcal{X} = \{a,b\}$.

• •$P(a)=0.99$, $P(b)=0.01$.
• •Consider two candidate models:
• •$Q₁(a)=0.9$, $Q₁(b)=0.1$ (covers both outcomes)
• •$Q₂(a)=0.9999$, $Q₂(b)=0.0001$ (very confident)

Forward KL $D_{\mathrm{KL}}(P\|\|Q)$ penalizes when $Q$ underestimates events that happen under $P$.

• •If $Q₂$ makes $b$ extremely unlikely but $P(b)=0.01$, the term $0.01\log\frac{0.01}{0.0001}$ can be significant.

Reverse KL $D_{\mathrm{KL}}(Q\|\|P)$ penalizes when $Q$ places mass where $P$ does not.

• •Here both support match (no zeros), but the weighting changes what matters: reverse KL is dominated by where $Q$ places mass (mostly at $a$), making it less sensitive to a small-probability region unless $Q$ also places mass there.

KL is not symmetric—and not meant to be

Thinking of KL as “how surprised would I be if I used Q in a world of P?” naturally picks a direction: the world is $P$, your model is $Q$.

So asymmetry is actually part of the meaning.

When does asymmetry show up in ML practice?

• •Supervised classification: cross-entropy corresponds to forward KL from the data label distribution to the model distribution.
• •Variational inference (ELBO): commonly minimizes reverse KL between an approximate posterior $q(z)$ and true posterior $p(z\mid x)$.
• •Distillation: direction determines whether the student covers teacher’s support broadly or focuses on peak probabilities.

Practical note: smoothing to avoid infinities

In discrete problems, if you estimate $Q$ from limited data, you can accidentally set $Q(x)=0$ for an event that later appears. Forward KL then becomes infinite.

Common fix: additive smoothing (Laplace/Dirichlet priors) so $Q(x)>0$ for all $x$.

This is not a hack; it encodes uncertainty and prevents “impossible” predictions from finite data artifacts.

Cross-entropy: the most common place you see KL

You already know entropy $H(P)$. Cross-entropy between $P$ and $Q$ is

And the identity

says: cross-entropy is entropy plus a mismatch penalty.

In classification, $P$ is often a one-hot (or soft) label distribution and $Q$ is the model’s predicted probabilities. Minimizing cross-entropy is minimizing KL (since $H(P)$ doesn’t depend on model parameters).

This is why KL is not just a theoretical object—it is embedded in the training objective of most neural classifiers.

Variational Autoencoders (VAEs): KL as a regularizer via ELBO

In VAEs, we introduce latent variables $z$ and want to maximize $\log p_\theta(x)$, but the posterior $p_\theta(z\mid x)$ is intractable.

We choose an approximate posterior $q_\phi(z\mid x)$ and maximize the ELBO:

Two important KL-related insights:

1. 1)The KL term is reverse KL (approx posterior to prior). It encourages $q_\phi(z\mid x)$ not to drift too far from $p(z)$.
2. 2)The ELBO can be derived by rewriting $\log p_\theta(x)$ and inserting $q_\phi$; the gap between $\log p_\theta(x)$ and ELBO is itself a KL:

So maximizing ELBO is minimizing a KL divergence to the true posterior.

This is a major “KL is everywhere” moment: it measures how far your approximation is from an intractable truth.

Information Bottleneck: KL as a constraint on information

The information bottleneck trades off:

• •how much information a representation $T$ retains about $X$
• •vs how much it preserves about target $Y$

Mutual information is defined via KL:

So KL is the primitive object beneath mutual information. The bottleneck objective can be written using KL-based quantities, and many derivations rely on KL manipulations.

A unifying mental model

Across these applications, KL is playing the same role:

• •There is a “reference truth” distribution (data labels, true posterior, joint distribution).
• •There is a “model/approximation” distribution.
• •KL measures the average log mismatch—the penalty you pay for pretending your approximation is real.

Visualization prompt: KL as a landscape to optimize

Canvas D: loss surface in parameter space

Take a simple family $Q_\theta$ (e.g., Bernoulli with parameter $q$) and a fixed target $P$ (Bernoulli with parameter $p$).

Plot $D_{\mathrm{KL}}(P\|\|Q_\theta)$ as a function of $q$.

Prompts:

• •Notice convexity in $q$ for Bernoulli forward KL.
• •Watch how gradients blow up near $q\to 0$ when $p>0$.

This links KL to optimization behavior: sometimes the objective is smooth and well-behaved; sometimes it has steep cliffs because of near-zero probabilities.

That optimization intuition is crucial when you later study:

• •numerical stability in softmax/cross-entropy
• •KL annealing in VAEs
• •mode collapse and distribution mismatch in generative models