Layer Normalization

The problem it solves (why before how)

Training a deep network means repeatedly composing nonlinear layers. Even if each layer is “reasonable,” their composition can drift into regimes where:

• •activations become very large or very small (numerical issues, saturation),
• •gradients become ill-conditioned (harder optimization),
• •different features in a layer live on very different scales (some features dominate updates),
• •training behavior changes when batch size changes.

Normalization layers attack these issues by standardizing activations so that the next computation sees inputs with a stable distribution.

Batch Normalization (BatchNorm) does this by computing statistics over the batch. That can be powerful, but it also means:

• •behavior depends on batch size,
• •training and inference behave differently (moving averages),
• •it’s awkward for variable-length sequences and autoregressive decoding,
• •it can be unstable for very small or micro-batches.

Layer Normalization (LayerNorm) is the alternative that makes normalization per example.

Intuition: “normalize the vector you have, not the batch you wish you had”

Take one training example flowing through a layer. At some point you have a feature vector x (for a single token, or a single hidden state, or a single fully-connected layer output). LayerNorm treats that vector as the object to normalize.

For an activation vector x ∈ ℝᵈ (d features), LayerNorm computes:

• •the mean across features (within that one example),
• •the variance across features (within that one example),
• •uses those to standardize each feature,
• •then restores flexibility by applying learned scale and shift.

The core definition

Let x = (x₁, x₂, …, x_d) be one example’s activation vector.

Per-example mean:

μ = (1/d) ∑ᵢ xᵢ

Per-example variance:

σ² = (1/d) ∑ᵢ (xᵢ − μ)²

Normalized activations:

x̂ᵢ = (xᵢ − μ) / √(σ² + ε)

Then an affine transform (learned per-feature):

yᵢ = γᵢ x̂ᵢ + βᵢ

Where:

• •γ (gamma) is a learned scale vector in ℝᵈ
• •β (beta) is a learned shift vector in ℝᵈ
• •ε is a small constant (e.g. 1e−5) to avoid division by 0

What it guarantees (and what it doesn’t)

If γ = 1 and β = 0, then for each example, the normalized vector x̂ has:

• •mean ≈ 0 across features
• •variance ≈ 1 across features

But LayerNorm does not make the distribution across the batch standardized, and it does not guarantee anything about each individual feature across the dataset. It’s “within-vector” normalization, not “within-feature-across-batch” normalization.

Where it’s applied

LayerNorm is used in:

• •Transformers (almost always)
• •RNNs / LSTMs (often)
• •MLP blocks in many modern architectures

A typical Transformer block normalizes hidden vectors of shape (batch, seq, d_model) across the last dimension d_model.

That last detail matters: LayerNorm normalizes across the feature dimension(s) you choose, usually the hidden size.

What exactly is being normalized?

Suppose you have a mini-batch of hidden states:

X ∈ ℝ^(B × d)

Row b is one example:

x^(b) ∈ ℝᵈ

LayerNorm computes statistics separately for each b.

For each example b:

μ^(b) = (1/d) ∑ᵢ xᵢ^(b)

σ²^(b) = (1/d) ∑ᵢ (xᵢ^(b) − μ^(b))²

Then:

x̂ᵢ^(b) = (xᵢ^(b) − μ^(b)) / √(σ²^(b) + ε)

Why normalize across features?

Think of a single layer output as a “feature descriptor” of the current example. If the overall magnitude and offset of that descriptor drift, subsequent layers must continuously re-adapt their weights to changing input scales.

LayerNorm reduces that burden by keeping each example’s feature vector in a normalized range.

A useful geometric view:

• •Subtracting μ removes the component along the all-ones direction (centering).
• •Dividing by √(σ² + ε) rescales so the average squared deviation is ~1.

This makes optimization more stable because the next layer sees inputs that are less sensitive to upstream shifts.

A careful note about variance definition

You may see variance computed with 1/d or 1/(d−1). In deep learning implementations, LayerNorm virtually always uses:

σ² = (1/d) ∑ᵢ (xᵢ − μ)²

This is not about unbiased estimation; it’s about a deterministic scaling that behaves consistently.

Multi-dimensional LayerNorm (common in practice)

In NLP / Transformers, your tensor might be:

X ∈ ℝ^(B × T × d)

You apply LayerNorm over the last dimension d:

For each (b, t), treat x^(b,t) ∈ ℝᵈ and normalize across i = 1…d.

In vision, you might have:

X ∈ ℝ^(B × C × H × W)

Depending on the variant, you might normalize across channels C only, or across C×H×W. “LayerNorm” in its original sense typically normalizes across the feature dimensions of a layer’s output; in convnets, that notion can blur into GroupNorm / InstanceNorm.

Numerical stability: why ε exists

If a vector’s features are all equal, then σ² = 0. Without ε you’d divide by 0.

More generally, if σ² is tiny, division can amplify noise. ε prevents extreme scaling:

√(σ² + ε) ≥ √ε

Typical ε values are 1e−5 or 1e−6.

BatchNorm vs LayerNorm: a comparison table

BatchNorm and LayerNorm are often confused because the formulas look similar. The key difference is which axes you average over.

Subtle but important: LayerNorm is deterministic given the example

BatchNorm introduces stochasticity because batch composition changes. LayerNorm’s stats for an example depend only on that example.

This is one reason LayerNorm plays nicely with autoregressive Transformers: during inference you often have batch size 1. BatchNorm would either be undefined or require stored running averages; LayerNorm just works.

What LayerNorm does to gradients (high-level)

You don’t need the full derivative to benefit from LayerNorm, but it helps to know what it changes:

• •It rescales gradients flowing into earlier layers because outputs are divided by √(σ² + ε).
• •It couples features: μ and σ² depend on all xᵢ, so each output yᵢ depends on all inputs xⱼ.

That coupling is intentional: the normalization is a property of the whole vector.

A conceptual takeaway:

• •Without normalization, one feature can explode and dominate the layer.
• •With LayerNorm, an exploding feature increases σ², which increases the denominator and dampens the standardized values.

Not a perfect “clamp,” but a stabilizing feedback.

Why normalize at all if it might remove useful information?

If we only output x̂, we force every example’s representation to be mean 0 and variance 1 across features. That sounds restrictive: what if a layer wants a particular feature to be systematically larger, or wants a nonzero mean for downstream computation?

This is why LayerNorm includes a learned per-feature affine transform:

y = γ ⊙ x̂ + β

Where:

• •γ, β ∈ ℝᵈ
• •⊙ is elementwise multiplication

The key idea: normalize for stability, then let the model “undo it” when beneficial

LayerNorm gives optimization stability, but γ and β let the network recover expressive power.

Two important facts:

1) If the best thing is “do nothing,” LayerNorm can learn that.

If γᵢ = √(σ² + ε) and βᵢ = μ (not exactly per-example, but conceptually), you can approximate reversing normalization. More practically: with learned γ and β, the network can set outputs to whatever scale and offset is useful.

2) γ can gate features.

If γᵢ becomes small, feature i’s normalized value contributes less. If γᵢ becomes large, that feature is amplified.

Why γ and β are per-feature (not scalars)

A single scalar γ would rescale the whole vector uniformly. But different features often need different typical magnitudes.

Per-feature γ, β allow:

• •one feature dimension to carry “large” signals,
• •another to stay small and precise,
• •shifting individual dimensions to match the distribution expected by subsequent weights.

Initialization choices

Common initializations:

• •γ initialized to 1 (so initial LayerNorm is close to pure normalization)
• •β initialized to 0

This makes the network start in a stable regime.

Where LayerNorm sits relative to residual connections (Pre-LN vs Post-LN)

In Transformers, you’ll see two popular placements:

• •Post-LN (original Transformer):
• •sublayer output is added to residual, then LayerNorm
• •pattern: h ← LN(h + Sublayer(h))

• •Pre-LN (common modern variant):
• •LayerNorm before the sublayer, residual added after
• •pattern: h ← h + Sublayer(LN(h))

Why this matters:

• •Pre-LN often improves gradient flow in very deep Transformers.
• •Post-LN makes the residual path go through a normalization, which can sometimes make optimization trickier for depth.

Either way, LayerNorm is still doing the same per-example feature normalization; the placement changes training dynamics.

A short derivation: verifying the normalized mean is 0 (when ε = 0)

Let x̂ᵢ = (xᵢ − μ)/σ with σ = √σ².

Compute mean across features:

(1/d) ∑ᵢ x̂ᵢ

= (1/d) ∑ᵢ (xᵢ − μ)/σ

= (1/σ) · (1/d) ∑ᵢ (xᵢ − μ)

= (1/σ) · ( (1/d) ∑ᵢ xᵢ − (1/d) ∑ᵢ μ )

= (1/σ) · ( μ − μ )

= 0

With ε > 0, the same algebra holds with σ = √(σ² + ε) (still a constant w.r.t. i), so the mean is still exactly 0.

Another derivation: normalized variance is ~1 (ε complicates slightly)

Variance across features:

(1/d) ∑ᵢ x̂ᵢ²

= (1/d) ∑ᵢ (xᵢ − μ)² / (σ² + ε)

= (1/(σ² + ε)) · (1/d) ∑ᵢ (xᵢ − μ)²

= (1/(σ² + ε)) · σ²

= σ² / (σ² + ε)

So if ε is tiny compared to σ², this is close to 1.

This explains why ε slightly reduces the achieved variance when σ² is small, which is exactly what we want for stability.

Practical shapes and parameter counts

If hidden size is d_model, then:

• •γ has d_model parameters
• •β has d_model parameters

LayerNorm adds 2d_model parameters per normalized vector space—usually negligible compared to attention/MLP weights.

Summary intuition

• •Normalization improves conditioning.
• •γ and β prevent the normalization from becoming a representational bottleneck.
• •Placement (pre/post) affects optimization but not the mathematical definition of LayerNorm.

Transformers: why LayerNorm fits the setting

Transformers process sequences with variable lengths, and often train with:

• •small effective batches (due to memory limits),
• •gradient accumulation (micro-batches),
• •distributed setups where per-device batch is tiny,
• •autoregressive inference with batch size 1.

BatchNorm depends on stable batch statistics. In these settings, batch statistics are noisy or unusable.

LayerNorm, by contrast:

• •uses per-example stats (independent of batch size),
• •behaves identically in training and inference,
• •naturally applies to each token’s hidden state vector.

Typical Transformer block usage

Let hidden states be H ∈ ℝ^(B × T × d).

A common (Pre-LN) block is:

1) U = LN(H)

2) A = Attention(U)

3) H ← H + Dropout(A)

4) V = LN(H)

5) M = MLP(V)

6) H ← H + Dropout(M)

LayerNorm is the “stabilizer” before each major transformation.

BatchNorm vs LayerNorm: choosing in practice

A simplified rule of thumb:

Important nuance: BatchNorm can act as regularization

BatchNorm’s batch-to-batch noise sometimes improves generalization.

LayerNorm is more “clean” and deterministic. If you lose that regularization benefit, you might rely more on:

• •dropout,
• •data augmentation,
• •weight decay,
• •stochastic depth.

LayerNorm and optimization stability (connection to conditioning)

Consider a linear layer:

z = W x + b

If x varies wildly in magnitude, then gradients w.r.t. W scale with x:

∂L/∂W includes terms proportional to x

So controlling the scale of x can stabilize updates. LayerNorm indirectly limits the variability of scales seen by downstream layers, improving the optimizer’s job.

When LayerNorm can be less ideal

LayerNorm is not universally best. Some considerations:

• •In convnets, normalizing across channels only (or groups) may better respect spatial structure.
• •In some reinforcement learning settings, normalization choices can interact with nonstationary data.
• •If you want batch-level noise benefits and have large batches, BatchNorm might still win.

Implementation mental model (no code, just steps)

For each example (and token position if sequence):

1) compute μ over the chosen feature axes

2) compute σ² over the same axes

3) normalize: (x − μ)/√(σ² + ε)

4) scale/shift: γ ⊙ (…) + β

Because the computation is local to each example, it parallelizes well and doesn’t require synchronization across devices the way BatchNorm can.

Connection forward: Transformers

Transformers rely on stable deep residual stacks of attention + MLP blocks. LayerNorm is one of the key pieces that makes those stacks trainable.

If you understand LayerNorm, you’re ready to reason about:

• •why “Pre-LN Transformers” train deeper,
• •why inference-time behavior is stable,
• •why normalization interacts with residual connections and learning rates.