Information Bottleneck

The problem IB is trying to solve (why before how)

In many learning settings, your input variable $X$ contains a mix of:

• •Signal: aspects of $X$ that help predict a target $Y$ (labels, future outcomes, etc.).
• •Nuisance: aspects of $X$ that are unrelated to $Y$ (noise, style, background, idiosyncrasies).

If you let a model store everything about $X$, it can overfit, memorize, or learn brittle features. If you compress too aggressively, you lose predictive power.

The Information Bottleneck (IB) framework turns this into a clean information-theoretic optimization:

• •Create a compressed representation $T$ of $X$.
• •Ensure $T$ retains information about $Y$.

Here $T$ is called the bottleneck variable because it limits how much information from $X$ can flow downstream.

The IB story in one line

Find a stochastic mapping $p(t|x)$ such that:

• •$T$ is as independent of $X$ as possible (compression)
• •while $T$ is as informative about $Y$ as possible (relevance)

The standard IB Lagrangian is:

Interpretation:

• •$I(X;T)$: how many bits $T$ “remembers” about $X$. Smaller means stronger compression.
• •$I(T;Y)$: how many bits $T$ carries about $Y$. Larger means better relevance.
• •$\beta$: trade-off knob.
• •Small $\beta$ → prioritize compression (small $I(X;T)$), risk losing predictive info.
• •Large $\beta$ → prioritize relevance (large $I(T;Y)$), allow more capacity.

A key modeling assumption: the Markov chain

IB usually assumes the representation is formed from $X$ alone:

This means: given $X$, $T$ is independent of $Y$ (because you compute $T$ from $X$).

Formally:

This assumption is not just a technicality—it encodes the idea that you don’t get to peek at the label $Y$ when forming the representation (at test time, you only see $X$).

“Compression” vs “relevance” as two competing forces

It helps to visualize two extremes:

IB doesn’t assume you know which parts of $X$ are relevant. It discovers them by optimizing these two pressures.

Pause and check 1 (sanity questions)

Before going further, make sure these answers feel clear:

1. 1)If $T$ is a copy of $X$, what is $I(X;T)$ likely to be? (Large—approximately $H(X)$.)
2. 2)If $T$ is constant (always the same value), what are $I(X;T)$ and $I(T;Y)$? (Both 0.)
3. 3)Why is $\beta$ needed? (Because you can’t generally minimize $I(X;T)$ and maximize $I(T;Y)$ simultaneously without trading them off.)

If those are intuitive, you’re ready for the mechanics.

Step 1: Start from what we control

We do not choose $p(x,y)$; it comes from the world/data. What we can choose is an encoder (possibly stochastic):

Together with $p(x,y)$, this induces:

• •$p(t) = \sum_x p(x) p(t|x)$
• •$p(y|t) = \sum_x p(y|x) p(x|t)$

Step 2: Expand the two mutual informations

Recall the mutual information identities:

And

The IB Lagrangian:

is therefore a functional of the entire conditional distribution $p(t|x)$.

What $I(X;T)$ means operationally

Think of transmitting $T$ given $X$. If $T$ contains many distinguishable states for different $x$, then $T$ carries many bits about $X$. If many different $x$ map to similar distributions over $t$, then $T$ forgets details.

A useful equivalent form is:

• •Increasing stochasticity in the encoder (larger $H(T|X)$) can reduce $I(X;T)$.
• •But if you make the encoder too noisy, you may also reduce $I(T;Y)$.

What $I(T;Y)$ means operationally

$T$ is “good” if it makes $Y$ predictable. Another identity:

Since $H(Y)$ is fixed by data, maximizing $I(T;Y)$ is equivalent to minimizing the conditional entropy $H(Y|T)$: make $Y$ as predictable from $T$ as possible.

The IB curve (rate–distortion flavor)

If you fix a compression budget $I(X;T) \le R$ and maximize relevance, you get a curve of optimal trade-offs. Equivalently, the Lagrangian with $\beta$ traces that curve.

This parallels rate–distortion theory:

• •Rate ↔ $I(X;T)$
• •Distortion ↔ a loss of predictive info about $Y$

IB can be seen as a task-aware compression method.

Pause and check 2 (micro-summary)

At this point:

• •We choose an encoder $p(t|x)$.
• •$I(X;T)$ penalizes how much $T$ remembers about $X$.
• •$I(T;Y)$ rewards how much $T$ helps predict $Y$.

Quick self-test:

• •If you increase $\beta$, do you expect $I(T;Y)$ to go up or down at the optimum? (Up.)
• •If you decrease $\beta$, do you expect $I(X;T)$ to go up or down? (Down.)

Now we’ll derive the structure of the optimal $p(t|x)$ in the discrete case.

Why fixed-point equations appear

The IB optimization is over distributions, not over a few parameters. In the discrete setting (finite $X$, $Y$, $T$), the classic IB solution gives self-consistent update rules for:

• •the encoder $p(t|x)$
• •the cluster prior $p(t)$
• •the decoder $p(y|t)$

This resembles a “soft clustering” procedure where each $x$ is assigned probabilistically to bottleneck states $t$.

Set up the constrained optimization

We minimize

subject to the normalization constraints:

We also rely on the Markov chain $T-X-Y$.

Key identity: relevance term becomes a KL to class-conditionals

A central result in IB is that the optimal encoder depends on how different $p(y|x)$ is from $p(y|t)$. The natural measure of “difference between predictive distributions” is KL divergence.

Define the KL:

Intuition: if $x$ and bottleneck state $t$ imply similar label distributions, then assigning $x$ to $t$ costs little relevance.

The fixed-point form (the classic IB equations)

The discrete IB solution satisfies:

1) Encoder update

where $Z(x,\beta)$ is a normalizer:

2) Cluster prior update

3) Decoder (label distribution per cluster)

and using Bayes:

Putting it together, (3) is often written as:

Show-your-work sketch: where does the exponential form come from?

We’ll outline the logic (not every calculus-of-variations detail, but the key steps).

Start with:

For the relevance term, using $I(T;Y)=\sum_{t}p(t)\sum_y p(y|t)\log\frac{p(y|t)}{p(y)}$.

When you take the functional derivative of $\mathcal{L}_{IB}$ with respect to $p(t|x)$ and enforce $\sum_t p(t|x)=1$ using Lagrange multipliers $\lambda(x)$, the stationary condition yields something of the form:

Exponentiate both sides:

and the proportionality constant is exactly $1/Z(x,\beta)$.

Intuition: IB as soft clustering of inputs by label-meaning

The encoder update says:

• •$p(t)$ favors common clusters.
• •The exponential penalizes assigning $x$ to clusters whose label-distribution $p(y|t)$ mismatches $p(y|x)$.
• •$\beta$ controls how “hard” the assignments become.

As $\beta \to 0$, the KL term is downweighted and assignments become dominated by $p(t)$ (compression wins).

As $\beta \to \infty$, the encoder strongly prefers clusters that match $p(y|x)$ (relevance wins), potentially making $T$ preserve nearly all predictive structure.

Pause and check 3 (sanity and pacing)

Answer these before moving on:

1. 1)In the encoder update, what happens if $p(y|x)=p(y|t)$ exactly? (KL=0, so $x$ can be assigned to $t$ with no relevance penalty.)
2. 2)Why does a KL divergence appear rather than, say, squared error? (Because we compare distributions over $Y$; KL is the natural information-theoretic discrepancy.)
3. 3)What makes these equations “fixed-point”? (Each update uses the others; iterating them seeks a self-consistent solution.)

Next we’ll shift from theory (discrete exact IB) to practice (continuous/high-dimensional settings).

Flagging the shift: why we need a variational approximation

The discrete IB fixed-point equations are elegant, but they assume we can:

• •Sum over all $x$, $t$, and $y$.
• •Represent $p(t|x)$ as a full table.
• •Compute $p(y|x)$ from the data distribution.

In modern ML:

• •$X$ is high-dimensional (images, text).
• •$T$ is continuous (latent vectors).
• •$p(y|x)$ is unknown; we only have samples.

So we use Variational Information Bottleneck (VIB): an optimization that resembles IB but is tractable with neural networks and minibatches.

The VIB setup

We parameterize:

• •Encoder $q_\phi(t|x)$ (neural net producing a distribution, often Gaussian)
• •Decoder/predictor $p_\theta(y|t)$ (classifier/regressor)
• •A prior over latents $p(t)$ (often standard normal)

Goal: keep $T$ predictive of $Y$ while limiting information from $X$ into $T$.

Replace the intractable terms with bounds

Relevance term

We want to maximize $I(T;Y)=H(Y)-H(Y|T)$. Since $H(Y)$ is constant, we minimize $H(Y|T)$.

In practice, we minimize negative log-likelihood under a decoder:

This is a standard prediction loss.

Compression term

The IB compression term is $I(X;T)$. In VIB, a common upper bound is:

Why this makes sense (sketch):

• •$I(X;T)=\mathbb{E}_{p(x)} D_{KL}(q(t|x)\|q(t))$.
• •$q(t)$ is the aggregated posterior, hard to compute.
• •Replacing $q(t)$ with a chosen prior $p(t)$ yields an upper bound.

The VIB objective (the practical loss)

Putting these together gives a common VIB training objective:

Compare to IB:

• •Prediction loss corresponds to maximizing relevance.
• •KL regularizer corresponds to compression.
• •$\beta$ again controls the trade-off.

Connection map: IB, β-VAE, and regularization

VIB looks structurally similar to other objectives:

Key difference: VIB uses $Y$ in the decoder; β-VAE reconstructs $X$.

Practical encoder choice: Gaussian + reparameterization

A common choice is:

Sampling is done via the reparameterization trick:

This makes gradients flow through stochastic sampling.

Why VIB can improve robustness and generalization

The KL term pushes $q_\phi(t|x)$ toward a simple prior (like $\mathcal{N}(0,I)$). That tends to:

• •limit memorization (capacity control)
• •encourage smoother representations
• •reduce sensitivity to nuisance variation in $X$

But if $\beta$ is too large, you get posterior collapse (the encoder ignores $X$ and $T$ carries little information), harming prediction.

Pause and check 4 (explicit bridge from theory to practice)

Make sure you can articulate these two transitions:

1. 1)Exact IB → VIB: We can’t compute $I(X;T)$ and $I(T;Y)$ exactly in deep settings, so we optimize a surrogate using NLL and a KL-to-prior.
2. 2)$p(t|x)$ → $q_\phi(t|x)$: We move from an unconstrained distribution to a parameterized family (neural network).

If that’s clear, the worked examples will feel grounded rather than magical.