Cross-Entropy

Why this concept exists

In information theory, we often ask: How many bits (or nats) do we need to encode outcomes?

• •If outcomes are truly generated from distribution p, then the best average code length you can hope for (with optimal coding) is the entropy:

H(p) = E_{x∼p}[−log p(x)]

• •But in modeling and machine learning, we often don’t know p. We build a model q and behave as if q were correct.

Cross-entropy answers the key practical question:

If reality follows p but I encode/predict using q, what is my average cost?

That cost is the cross-entropy:

H(p,q) = E_{x∼p}[−log q(x)]

So cross-entropy is the expected negative log-probability that the model q assigns to samples drawn from the true distribution p.

Intuition: “average surprise under q when the world follows p”

When q(x) is small for outcomes that happen often under p, −log q(x) is large. Cross-entropy accumulates these penalties in proportion to how often the world produces each outcome.

• •If q matches p perfectly, then q(x) = p(x) and cross-entropy becomes entropy:

H(p,p) = E_{x∼p}[−log p(x)] = H(p)

• •If q is wrong in places that matter (where p is large), cross-entropy increases.

Discrete vs continuous (important nuance)

For a discrete variable X taking values x ∈ 𝒳:

H(p,q) = −∑_{x∈𝒳} p(x) log q(x)

For continuous variables with densities p(x), q(x), we typically talk about cross-entropy of densities:

H(p,q) = E_{x∼p}[−log q(x)] = −∫ p(x) log q(x) dx

This behaves like differential entropy: it can be negative and depends on coordinate transformations. In ML classification, we’re nearly always in the discrete case (finite classes), so the discrete formula is the main one here.

Units: bits vs nats

The log base sets the unit.

Most ML libraries use natural log, so cross-entropy is in nats.

Why cross-entropy is a loss

A loss should be:

1) Small when predictions are good

2) Large when predictions are confidently wrong

3) Smooth enough to optimize (typically differentiable)

Cross-entropy does all three:

• •If q assigns high probability to what occurs, −log q is small.
• •If q assigns near-zero probability to what occurs, −log q blows up.
• •With common parameterizations (like softmax), it has clean gradients.

This makes cross-entropy the workhorse loss for probabilistic classification.

Why this decomposition matters

When training a model, we want q to get closer to p. But p is unknown. So we need a criterion that:

• •can be estimated from samples from p (the dataset)
• •leads to a principled notion of “closeness”

The key identity:

H(p,q) = H(p) + KL(p‖q)

tells you cross-entropy equals:

• •an irreducible term H(p) (uncertainty inherent in the data-generating process), plus
• •a model mismatch term KL(p‖q)

Since H(p) doesn’t depend on q, minimizing cross-entropy over q is equivalent to minimizing KL(p‖q).

Derivation (discrete case)

Start from definitions:

H(p,q) = −∑ p(x) log q(x)

H(p) = −∑ p(x) log p(x)

KL(p‖q) = ∑ p(x) log (p(x)/q(x))

Now expand KL:

KL(p‖q)

= ∑ p(x) [log p(x) − log q(x)]

= ∑ p(x) log p(x) − ∑ p(x) log q(x)

Multiply by −1 where useful:

−∑ p(x) log q(x) = −∑ p(x) log p(x) + KL(p‖q)

Recognize the left side as cross-entropy and the first term as entropy:

H(p,q) = H(p) + KL(p‖q)

Consequences you should internalize

1) Lower bound: since KL(p‖q) ≥ 0,

H(p,q) ≥ H(p)

and equality holds iff p = q (almost everywhere).

2) What training can and cannot do: even with a perfect model family and infinite data, the best achievable cross-entropy is H(p). You cannot beat the intrinsic randomness of the world.

3) “Cross-entropy minimization” is KL minimization: many ML training objectives are really KL projections:

q* = argmin_q H(p,q)

= argmin_q KL(p‖q)

This is the information-theoretic justification for maximum likelihood and many probabilistic learning procedures.

Estimating cross-entropy from data

In practice, we don’t know p, but we have samples x₁,…,xₙ ∼ p. Then:

H(p,q) = E_{x∼p}[−log q(x)]

is estimated by the empirical mean:

Ĥ(p,q) = (1/n) ∑_{i=1}^n [−log q(xᵢ)]

This is exactly the negative log-likelihood (NLL) per sample. Minimizing it is maximum likelihood estimation.

Relationship to log-loss

In binary classification and multiclass classification, cross-entropy is often called:

• •log-loss
• •negative log-likelihood

They differ mostly in framing; mathematically they are the same objective (up to averaging vs summing).

A small but important support condition

If there exists x with p(x) > 0 but q(x) = 0, then −log q(x) is infinite, so:

H(p,q) = ∞

This is a feature, not a bug: assigning zero probability to an event that occurs is maximally bad under log-loss. In practice, this motivates smoothing, label smoothing, or ensuring q has full support (e.g., softmax always yields positive probabilities).

Why cross-entropy fits classification so well

Classification asks for probabilities over classes. If the true label is y, you want the model to assign high probability q(y|x). Cross-entropy turns that into a scalar penalty:

ℓ(x,y) = −log q(y|x)

This is exactly the penalty you’d derive from maximum likelihood for a categorical distribution.

Setting: K-class classification

Let classes be {1,…,K}.

• •True conditional distribution: p(y|x)
• •Model predicted distribution: q(y|x)

For a single x, the conditional cross-entropy is:

H(p(·|x), q(·|x)) = −∑_{k=1}^K p(k|x) log q(k|x)

One-hot labels

In supervised classification, we typically observe a single label y for each x, and we treat the “true” distribution as one-hot:

p(k|x) = 1 if k = y, else 0

Plugging in:

H(p(·|x), q(·|x))

= −∑_{k=1}^K p(k|x) log q(k|x)

= −log q(y|x)

So the multiclass cross-entropy loss per example is just the negative log probability of the correct class.

Softmax parameterization (where gradients become simple)

A common model outputs logits z ∈ ℝ^K (bold because it’s a vector). Softmax maps logits to probabilities:

q(k|x) = softmax(z)_k = exp(z_k) / ∑_{j=1}^K exp(z_j)

Then the loss for a one-hot label y is:

ℓ = −log q(y|x)

= −log \left( \frac{exp(z_y)}{∑_{j=1}^K exp(z_j)} \right)

Now simplify:

ℓ

= −[z_y − log(∑_{j=1}^K exp(z_j))]

= −z_y + log(∑_{j=1}^K exp(z_j))

This form is numerically and conceptually important.

Gradient of softmax cross-entropy (the famous “q − y” result)

Why do practitioners love this loss? Because the derivative is clean.

Let p̂ be the one-hot vector for the true class (so p̂_k = 1{k=y}). Let q be softmax(z).

For each class k,

∂ℓ/∂z_k = q_k − p̂_k

Sketch of the derivation (enough steps to see it):

ℓ = −z_y + log(∑_j exp(z_j))

For k ≠ y:

∂ℓ/∂z_k

= 0 + (1/(∑_j exp(z_j))) · exp(z_k)

= exp(z_k)/∑_j exp(z_j)

= q_k

For k = y:

∂ℓ/∂z_y

= −1 + exp(z_y)/∑_j exp(z_j)

= q_y − 1

Combine using p̂_k:

∂ℓ/∂z_k = q_k − p̂_k

This says: update is proportional to (predicted probability − target probability).

Binary cross-entropy as a special case

For y ∈ {0,1}, and model probability q = σ(z) where σ is sigmoid:

σ(z) = 1/(1+exp(−z))

Binary cross-entropy per sample:

ℓ = −[ y log q + (1−y) log(1−q) ]

This matches Bernoulli negative log-likelihood.

Calibration and confidence

Cross-entropy does more than choose the right class—it pushes the predicted probability on the true class toward 1. That can yield well-calibrated probabilities when the model is correct and not overly flexible, but also encourages high confidence (which can be problematic under dataset shift).

A quick comparison:

Cross-entropy is stricter than “just being correct”: predicting 0.51 on the true class is “correct” but still heavily penalized compared to 0.99.

Label smoothing (a direct modification of p)

Sometimes we don’t want one-hot targets because they make the model overconfident. Label smoothing replaces p̂ with a softened distribution:

p̃ = (1−ε)·p̂ + ε·u

where u is uniform (u_k = 1/K).

Then the loss is:

ℓ = −∑_{k=1}^K p̃_k log q_k

This prevents assigning infinite penalty-like gradients to small mistakes and often improves generalization and calibration.

Maximum likelihood estimation (MLE) is cross-entropy minimization

Suppose you have a dataset D = {x₁,…,xₙ} drawn i.i.d. from p, and a parametric model q_θ.

The log-likelihood is:

log L(θ) = ∑_{i=1}^n log q_θ(xᵢ)

Maximizing log-likelihood is equivalent to minimizing negative log-likelihood:

NLL(θ) = −∑_{i=1}^n log q_θ(xᵢ)

Divide by n:

(1/n) NLL(θ) = (1/n) ∑ −log q_θ(xᵢ)

As n → ∞, this converges (under standard assumptions) to:

E_{x∼p}[−log q_θ(x)] = H(p, q_θ)

So MLE is choosing θ to minimize cross-entropy from p to q_θ.

Perplexity: exponentiated cross-entropy

In language modeling and other discrete-sequence models, a standard metric is perplexity.

If logs are natural:

perplexity = exp(H)

If logs are base 2:

perplexity = 2^{H}

Interpretation: perplexity is the “effective number of equally likely choices” the model faces. Lower perplexity means the model is less surprised.

Cross-entropy rate for sequences

For sequences x₁:T, models often factorize:

q(x₁:T) = ∏_{t=1}^T q(x_t | x_{<t})

Then:

−log q(x₁:T) = −∑_{t=1}^T log q(x_t | x_{<t})

Average per token cross-entropy:

(1/T) ∑_{t=1}^T [−log q(x_t | x_{<t})]

This is exactly what token-level training minimizes.

Proper scoring rule (why it encourages honesty)

Cross-entropy / log-loss is a strictly proper scoring rule: the expected score is minimized when you predict the true distribution.

Informally: if you can choose q, and your goal is to minimize E_{x∼p}[−log q(x)], the best choice is q = p.

That’s a deep reason it is preferred over ad-hoc losses for probabilistic prediction.

Numerical stability: log-sum-exp trick

Softmax cross-entropy often uses:

ℓ = −z_y + log(∑_j exp(z_j))

Directly computing exp(z_j) can overflow if z_j is large. The standard trick subtracts m = max_j z_j:

log(∑_j exp(z_j))

= log(∑_j exp(z_j − m) · exp(m))

= m + log(∑_j exp(z_j − m))

This keeps numbers stable.

When cross-entropy is the wrong tool

Cross-entropy assumes your labels represent draws from a categorical/Bernoulli distribution and that you care about probabilistic accuracy.

It can be mismatched when:

• •labels are extremely noisy and you only want ranking
• •you care about margins more than probabilities
• •evaluation metric is non-probabilistic (e.g., top-k accuracy, F1) and calibration is irrelevant

Even then, cross-entropy is often a good default because it is smooth and statistically grounded, but it’s worth recognizing the assumption.

Connection back to information theory

The identity

H(p,q) = H(p) + KL(p‖q)

says: your average code length using code optimized for q equals the optimal code length (H(p)) plus an inefficiency penalty (KL). In ML terms: you can’t remove inherent uncertainty, but you can remove mismatch by improving q.

This “coding view” is not just philosophy—it’s why log-loss is so ubiquitous across compression, density estimation, and modern neural modeling.