Variance

Why we need more than an average

Expected value tells you the center of a distribution: μ = E[X]. But two random variables can have the same μ and behave very differently.

Example intuition:

• •X is always 0.
• •Y is +10 half the time and −10 half the time.

Both have E[X] = E[Y] = 0, but Y “swings” wildly while X never moves. Variance is designed to capture that swing.

Definition

Let X be a random variable with mean μ = E[X]. The variance of X is

Var(X) = E[(X − μ)²].

Read this as: “take the deviation from the mean, square it, then average.”

Why squared deviation?

We want a measure of spread that:

1. 1)Treats positive and negative deviations symmetrically (so they don’t cancel).
2. 2)Penalizes larger deviations more than smaller ones.
3. 3)Plays nicely with algebra and optimization (this becomes important in statistics and ML).

Absolute deviation |X − μ| would avoid cancellation too, but squaring is the choice that leads to particularly useful identities (like Var(X) = E[X²] − μ²) and makes many derivations clean.

Units and standard deviation

If X has units (meters, dollars, etc.), then (X − μ)² has squared units, so variance has squared units too. To get back to the original units, define the standard deviation:

σ = √Var(X).

Variance is the main mathematical object; standard deviation is often easier to interpret in the original scale.

A quick geometric picture

If you think of outcomes x as points on a line, μ is the “balance point,” and (x − μ)² is the squared distance to that point. Variance is the expected squared distance to the center.

The direct computation

From the definition, for a discrete random variable:

Var(X) = E[(X − μ)²] = ∑ (x − μ)² P(X = x).

This is conceptually simple:

1. 1)Find μ.
2. 2)For each outcome x, compute (x − μ)².
3. 3)Weight by P(X = x) and sum.

For a continuous random variable with density f(x), replace the sum with an integral:

Var(X) = ∫ (x − μ)² f(x) dx.

The shortcut identity: Var(X) = E[X²] − (E[X])²

Computing (x − μ)² can be tedious. There’s a standard algebraic identity that makes variance faster to compute.

Start with the definition and expand:

Var(X)

= E[(X − μ)²]

= E[X² − 2μX + μ²].

Now use linearity of expectation and the fact that μ is a constant:

E[X² − 2μX + μ²]

= E[X²] − 2μE[X] + E[μ²]

= E[X²] − 2μ·μ + μ²

= E[X²] − μ².

So:

Var(X) = E[X²] − (E[X])².

This is one of the most-used formulas in probability and statistics.

Why this identity matters

It separates variance into two pieces you can often compute easily:

• •E[X] (first moment)
• •E[X²] (second moment)

Many common distributions have known formulas for these moments, making Var(X) almost immediate.

Invariance to shifts (adding a constant)

A key property: shifting a random variable by a constant does not change its variance.

Let Y = X + c. Then E[Y] = E[X] + c = μ + c.

Compute variance:

Var(Y)

= E[(Y − E[Y])²]

= E[(X + c − (μ + c))²]

= E[(X − μ)²]

= Var(X).

So variance measures spread, not location.

Scaling rule (multiplying by a constant)

If you scale outcomes, spread scales quadratically:

Let Y = aX. Then E[Y] = aμ.

Var(Y)

= E[(aX − aμ)²]

= E[a²(X − μ)²]

= a²E[(X − μ)²]

= a²Var(X).

So:

Var(aX) = a²Var(X).

This is why standard deviation scales linearly: √Var(aX) = |a| √Var(X).

Quick reference table

Variance is always nonnegative

From the definition:

Var(X) = E[(X − μ)²].

But (X − μ)² ≥ 0 for every outcome, because a square can’t be negative. An expectation of a nonnegative random variable is nonnegative, so:

Var(X) ≥ 0.

This is not just a technicality: it tells you variance behaves like a “size” or “energy.”

When is variance exactly zero?

Variance is zero exactly when there is no spread at all.

Claim:

Var(X) = 0 ⇔ X = μ almost surely.

Meaning: X equals the constant μ with probability 1 (it may differ on a probability-0 set, which doesn’t affect expectations).

Why “if” direction is true

If X = μ (with probability 1), then X − μ = 0 (with probability 1), so (X − μ)² = 0 always. Therefore E[(X − μ)²] = 0.

Why “only if” direction is true

If Var(X) = E[(X − μ)²] = 0 and (X − μ)² is always ≥ 0, the only way the average can be 0 is if it is 0 wherever probability mass exists.

Formally (intuition-first):

• •If there were any positive probability that (X − μ)² > 0, then the expectation would be > 0.
• •So it must be that P((X − μ)² = 0) = 1, i.e., P(X = μ) = 1.

“Almost surely” in plain terms

“Almost surely” (a.s.) means “with probability 1.”

It allows edge cases where X might misbehave on events of probability 0, but those events do not affect expectations, variances, or most practical probability calculations.

Example:

• •A continuous random variable might have P(X = 0) = 0, but it could still be possible that some definition changes at x = 0 without affecting anything measurable.

For variance, “Var(X) = 0 implies X is constant a.s.” is the correct statement.

Variance as a measure of concentration

The smaller Var(X) is, the more tightly X concentrates around μ.

A very important inequality (you don’t need to master it yet, but it motivates why variance matters) is Chebyshev’s inequality:

P(|X − μ| ≥ kσ) ≤ 1/k².

It says: if variance is small, large deviations from the mean are unlikely.

This is a major reason variance is used as a compact summary of uncertainty.

Variance in common distributions

Once you know how to compute variance, you can summarize a distribution with (mean, variance). Many families are parameterized this way.

A few examples you’ll meet soon:

• •Bernoulli(p): mean p, variance p(1 − p)
• •Binomial(n, p): mean np, variance np(1 − p)
• •Poisson(λ): mean λ, variance λ
• •Normal(μ, σ²): mean μ, variance σ²

Variance is the “second parameter” in many models because it controls spread.

From variance to covariance and correlation

Variance is the special case of covariance when you compare a variable with itself.

Cov(X, Y) is defined as:

Cov(X, Y) = E[(X − E[X])(Y − E[Y])].

If you plug in Y = X:

Cov(X, X) = E[(X − E[X])(X − E[X])]

= E[(X − μ)²]

= Var(X).

So variance is “self-covariance.” This viewpoint is crucial later:

• •Covariance matrices describe how a vector random variable X (boldface often used for vectors) varies in multiple dimensions.
• •The diagonal entries are variances of each component.

Correlation normalizes covariance by standard deviations:

Corr(X, Y) = Cov(X, Y) / (σₓ σᵧ).

Variance is the building block for both.

Why ML cares: normalization and training stability

In deep learning, activations can drift to have large or tiny variance across layers. If variance explodes or vanishes, gradients can become unstable.

Normalization methods (BatchNorm, LayerNorm) explicitly use mean and variance to re-center and re-scale activations.

A typical pattern for a batch of activations a:

1. 1)Compute mean μ̂
2. 2)Compute variance v̂
3. 3)Normalize: (a − μ̂) / √(v̂ + ε)

This makes the normalized activations have approximately mean 0 and variance 1.

Even if you’re not training neural nets yet, the message is:

• •Variance is not just descriptive statistics.
• •It is a control knob for stability in algorithms.

Population variance vs sample variance (preview)

In probability, Var(X) is a population quantity: it assumes you know the true distribution.

In statistics, you estimate variance from samples x₁, …, xₙ. Two common formulas are:

• •“Biased” estimator: (1/n) ∑(xᵢ − x̄)²
• •“Unbiased” estimator: (1/(n−1)) ∑(xᵢ − x̄)²

You don’t need the full derivation here, but it’s helpful to recognize that variance connects probability theory (true Var) to data analysis (estimated variance).