Loss Functions

Why we need a loss at all

A learning algorithm needs a feedback signal: given a prediction ŷ and a true target y, how bad was the prediction? In machine learning we usually want something we can:

1. 1)Compute per example (so we can aggregate over a dataset)
2. 2)Differentiate (so we can update parameters θ via gradients)
3. 3)Optimize (ideally with stable, predictable behavior)

A pointwise loss is a scalar function

L(y, ŷ) ∈ ℝ

that measures the error on a single example.

Given a dataset {(xᵢ, yᵢ)}ᵢ₌₁ⁿ and a model f(x; θ) producing ŷᵢ = f(xᵢ; θ), the training objective is typically the empirical risk:

R̂(θ) = (1/n) ∑ᵢ L(yᵢ, f(xᵢ; θ))

This turns learning into an optimization problem:

θ* = argmin_θ R̂(θ)

Loss vs metric (don’t conflate them)

A metric is what you care about at evaluation time (accuracy, F1, BLEU, AUC, etc.). A loss is what you minimize during training. They often differ because many metrics are:

• •discontinuous (accuracy jumps when you cross a threshold)
• •non-differentiable
• •unstable for gradient descent

Loss functions are usually surrogates: smooth (or piecewise smooth) objectives that correlate with the metric but are easier to optimize.

The “shape” of a loss encodes behavior

Losses don’t just say “wrong or right”; they encode how wrong. For example:

• •Squared error heavily punishes large mistakes.
• •Cross-entropy punishes confident wrong probabilities extremely.
• •Hinge loss encourages a margin, not just correctness.

The choice of loss is therefore a modeling decision, not a minor detail.

Differentiability and subdifferentiability

Most modern training uses gradients. For that, we want ∂L/∂ŷ to exist (at least almost everywhere). Some useful losses are not differentiable everywhere (e.g., hinge, absolute error). For convex piecewise-linear losses, we can use subgradients.

A quick mental model:

• •Differentiable: you can compute a gradient ∇θ R̂(θ) and use gradient descent/Adam.
• •Subdifferentiable: you can compute a subgradient and still optimize with subgradient methods (or autodiff handles piecewise definitions).

Even in deep learning, many “non-smooth” losses are fine because the non-differentiable points form a set of measure zero; optimization typically proceeds with valid gradients almost everywhere.

Why this decomposition matters

It’s tempting to see training as “minimize a big function.” In practice, the big function is built from many small ones.

• •The pointwise loss L(yᵢ, ŷᵢ) tells you how one example contributes.
• •The empirical risk aggregates these contributions.

This structure enables mini-batch training: you estimate the full gradient using a small subset of examples.

From loss to gradient: the chain rule pipeline

Suppose the model output is ŷ = f(x; θ). Then

R̂(θ) = (1/n) ∑ᵢ L(yᵢ, f(xᵢ; θ))

Differentiate w.r.t. θ:

∇θ R̂(θ)

= ∇θ ( (1/n) ∑ᵢ L(yᵢ, f(xᵢ; θ)) )

= (1/n) ∑ᵢ ∇θ L(yᵢ, f(xᵢ; θ))

Now apply the chain rule for each term:

∇θ L(yᵢ, f(xᵢ; θ))

= (∂L/∂ŷᵢ) · ∇θ f(xᵢ; θ)

So losses matter because they determine ∂L/∂ŷ, the “error signal” that gets backpropagated.

Mini-batches: stochastic empirical risk

For a mini-batch B of size m, we use:

R̂_B(θ) = (1/m) ∑ᵢ∈B L(yᵢ, f(xᵢ; θ))

and update parameters using ∇θ R̂_B(θ). This is an unbiased estimate of the full gradient when B is sampled uniformly.

Output type drives loss choice

Before picking a loss, identify what ŷ represents:

• •Regression: ŷ ∈ ℝ (or ℝᵈ)
• •Binary probability: ŷ ∈ (0, 1)
• •Multiclass probability: p̂ ∈ Δᵏ (simplex: p̂ⱼ ≥ 0, ∑ⱼ p̂ⱼ = 1)
• •Logits: z ∈ ℝᵏ (unnormalized scores); softmax converts logits to probabilities

Loss functions are often defined in terms of probabilities, but implemented with logits for numerical stability.

A comparison table (what you optimize changes what you get)

The same dataset can yield qualitatively different models depending on whether you optimize probability calibration (cross-entropy) or margins (hinge).

1) Mean Squared Error (MSE)

Why MSE is so common

MSE makes sense when:

• •targets are real-valued
• •noise is roughly Gaussian
• •you want to punish large deviations strongly

Pointwise MSE is often written with a 1/2 for cleaner derivatives:

L(y, ŷ) = (1/2)(ŷ − y)²

Derivative wrt prediction:

∂L/∂ŷ

= ∂/∂ŷ [ (1/2)(ŷ − y)² ]

= (1/2) · 2(ŷ − y)

= (ŷ − y)

This is a very clean error signal: “prediction minus truth.”

For vector-valued regression y ∈ ℝᵈ, you often use:

L(y, ŷ) = (1/2)‖ŷ − y‖²

with gradient:

∇_{ŷ} L = ŷ − y

What can go wrong

If your data has heavy-tailed noise or outliers, squared error can over-focus on a few extreme points. A robust alternative is MAE or Huber loss.

2) Cross-Entropy (Log Loss)

You already know cross-entropy conceptually (H(p, q) = H(p) + KL(p‖q)). Here we connect it directly to the training loss and its gradients.

Binary cross-entropy (BCE)

For y ∈ {0, 1} and predicted probability p̂ ∈ (0, 1):

L(y, p̂) = −[ y log(p̂) + (1 − y) log(1 − p̂) ]

This is the negative log-likelihood of a Bernoulli model.

If the model outputs a logit z ∈ ℝ with p̂ = σ(z) = 1/(1 + e^(−z)), the BCE gradient wrt z has a beautifully simple form:

First, note:

∂L/∂p̂ = −( y/p̂ − (1 − y)/(1 − p̂) )

and

∂p̂/∂z = p̂(1 − p̂)

Chain rule:

∂L/∂z

= (∂L/∂p̂)(∂p̂/∂z)

= −( y/p̂ − (1 − y)/(1 − p̂) ) · p̂(1 − p̂)

= −( y(1 − p̂) − (1 − y)p̂ )

= p̂ − y

So for logistic regression / binary classification:

∂L/∂z = p̂ − y

This mirrors MSE’s “prediction minus truth,” but in probability space.

Multiclass cross-entropy

Let y be a one-hot vector over K classes, and p̂ = softmax(z) where:

p̂ⱼ = exp(zⱼ) / ∑ₖ exp(zₖ)

Cross-entropy loss:

L(y, p̂) = −∑ⱼ yⱼ log(p̂ⱼ)

If y is one-hot with true class c, this simplifies to:

L = −log(p̂_c)

A key gradient identity (used constantly in deep learning):

∂L/∂zⱼ = p̂ⱼ − yⱼ

That is: softmax + cross-entropy yields a stable gradient of “predicted distribution minus target distribution.”

Geometry: why it punishes confident mistakes

If the true class is c but p̂_c is tiny, then −log(p̂_c) is huge. That’s exactly the point: the loss is extremely sensitive to confident wrong predictions.

3) Hinge Loss (Margin-Based)

Why hinge exists

Sometimes you don’t care about calibrated probabilities; you care about a decision boundary with a safety buffer. Hinge loss is a convex surrogate for 0–1 classification loss that promotes a margin.

For binary labels y ∈ {−1, +1} and a score s (often s = wᵀx + b):

L(y, s) = max(0, 1 − y s)

Interpretation:

• •If y s ≥ 1, the point is correctly classified with margin ≥ 1 → loss 0
• •If 0 < y s < 1, correct but too close to boundary → positive loss
• •If y s ≤ 0, misclassified → even larger loss

Subgradient

Hinge is not differentiable at y s = 1, but it is subdifferentiable.

For the score s:

If 1 − y s < 0 ⇒ L = 0 ⇒ ∂L/∂s = 0

If 1 − y s > 0 ⇒ L = 1 − y s ⇒ ∂L/∂s = −y

At 1 − y s = 0 ⇒ subgradient set includes values between 0 and −y

This piecewise behavior is what creates the “support vectors”: only points violating the margin contribute gradients.

Choosing between cross-entropy and hinge (a practical comparison)

Cross-entropy usually wins in deep learning because it works naturally with probabilistic outputs and backprop, but hinge can be useful when margin is the primary concern.

Why custom losses are common

Real tasks rarely match “plain regression” or “plain classification.” You may have:

• •class imbalance
• •different costs for false positives vs false negatives
• •multiple objectives (accuracy + fairness + latency)
• •structured outputs (sequences, boxes, graphs)
• •a metric that is non-differentiable (F1, IoU, BLEU)

Custom losses are how you encode these realities into optimization.

Weighted losses (cost-sensitive learning)

Suppose binary classification where positive examples are rare. A standard BCE might be minimized by predicting “negative” too often.

Weighted BCE:

L(y, p̂) = −[ α y log(p̂) + β (1 − y) log(1 − p̂) ]

• •α > β emphasizes recall on positives
• •β > α emphasizes precision / avoiding false positives

For multiclass, use class weights w_c:

L = − w_c log(p̂_c)

This is simple and effective, but you must tune weights carefully; extreme weights can destabilize training.

Label smoothing (a small tweak, big effect)

Instead of one-hot targets, use a softened target distribution:

ỹⱼ = (1 − ε) yⱼ + ε / K

Then train with cross-entropy to ỹ. Benefits:

• •reduces overconfidence
• •improves calibration
• •can improve generalization

It changes the gradient target from y to ỹ, preventing the model from pushing logits to extreme values.

Robust regression: Huber loss

To interpolate between MSE (sensitive) and MAE (robust), use Huber with threshold δ:

Let r = ŷ − y.

L(r) =

• •(1/2) r² if |r| ≤ δ
• •δ(|r| − (1/2)δ) if |r| > δ

Derivative wrt ŷ (i.e., wrt r):

∂L/∂ŷ =

• •r if |r| ≤ δ
• •δ · sign(r) if |r| > δ

So small errors behave like MSE, but large errors have bounded influence like MAE.

Multi-task losses: weighted sums

If your model predicts multiple outputs (say classification + bounding box regression), you may use:

L_total = λ₁ L₁ + λ₂ L₂

This is easy to write, but hard to tune. If λ₂ is too large, the model may ignore task 1.

A useful checklist:

• •Are losses on comparable scales?
• •Do gradients from each task have similar magnitudes?
• •Do you need dynamic weighting (learned λ’s or normalization)?

When the metric is non-differentiable

Many “true” metrics are not differentiable. Common strategies:

1. 1)Surrogate loss (most common)

• •Use cross-entropy instead of accuracy
• •Use smooth IoU approximations for segmentation

2. 2)Reinforcement learning / policy gradients

• •Optimize expected reward E[R]
• •Used in sequence generation and RLHF

3. 3)Direct search / black-box optimization

• •Not typical for deep nets, but used in some hyperparameter or prompt tuning loops

Numerical stability: implement losses carefully

Cross-entropy with softmax can overflow if you compute exp(z) naively. In practice you compute:

log softmax(z_c) = z_c − log(∑ⱼ exp(zⱼ))

using the log-sum-exp trick:

log(∑ⱼ exp(zⱼ))

= a + log(∑ⱼ exp(zⱼ − a))

where a = maxⱼ zⱼ.

Similarly, binary cross-entropy is best computed from logits directly (many libraries provide a “BCEWithLogits” function).

Connection to empirical risk minimization (ERM)

At a high level, “training” is ERM: minimize R̂(θ). The loss is your stand-in for what you actually care about.

A good loss should be:

• •aligned with the task goal (or a good surrogate)
• •stable for optimization (smooth enough, well-scaled gradients)
• •consistent with output interpretation (probabilities vs scores)
• •compatible with your optimizer (differentiable or subdifferentiable)

Loss choice is one of the few levers that directly changes what gradients you get, and therefore what model you end up with.