Covariance and Correlation

The motivation (why we need them)

Variance tells you how a single random variable spreads around its mean. But many real questions are about pairs of variables:

• •Do study hours and exam score tend to rise together?
• •Do two sensors drift together over time?
• •If one stock goes up, does another tend to go up as well?

We want a number that captures co-movement.

Covariance: “do deviations move together?”

Covariance looks at whether $X$ and $Y$ are simultaneously above their means or simultaneously below their means.

Define the mean-centered variables:

• •$X_c = X - E[X]$
• •$Y_c = Y - E[Y]$

Then covariance is the expected product:

Interpretation by sign:

• •Positive covariance: when $X$ is above its mean, $Y$ tends to be above its mean too (and likewise below).
• •Negative covariance: when $X$ is above its mean, $Y$ tends to be below its mean.
• •Near zero covariance: no linear tendency for the deviations to align.

A key subtlety: covariance is unitful. If $X$ is in dollars and $Y$ is in seconds, covariance is in dollar·seconds. Changing units (e.g., dollars → cents) scales covariance.

Correlation: “covariance without units”

To compare relationships across different scales, we standardize by each variable’s standard deviation.

where $\sigma_X = \sqrt{\operatorname{Var}(X)}$ and $\sigma_Y = \sqrt{\operatorname{Var}(Y)}$.

Correlation is:

• •unitless
• •bounded: $-1 \le \rho_{X,Y} \le 1$

Interpretation:

• •$\rho = 1$: perfect positive linear relationship ($Y = aX + b$ with $a>0$ almost surely)
• •$\rho = -1$: perfect negative linear relationship ($a<0$)
• •$\rho = 0$: no linear association (but there may be a nonlinear one)

A concrete picture: centered scatter and “cosine similarity”

If you take paired samples $(xᵢ, yᵢ)$ and center them, each data point becomes a 2D vector from the mean: $\mathbf{z}_i = [xᵢ-\bar{x},\; yᵢ-\bar{y}]$.

A helpful geometric intuition is that correlation behaves like a normalized “alignment” between the centered coordinates of $X$ and $Y$.

Here’s a simple ASCII scatter showing a positive association. The “+” is the mean; arrows suggest centered deviations.

Y
^               *
|            *
|         *
|      *
|   *
| +-----------------> X
|   *
|      *

When points trend from bottom-left to top-right, products $(xᵢ-\bar{x})(yᵢ-\bar{y})$ are often positive, boosting covariance and correlation.

What they can and cannot conclude (preview)

Covariance/correlation can:

• •quantify direction and strength of linear relationship
• •feed into models (regression) and decompositions (PCA)

They cannot by themselves:

• •prove causation (ice cream doesn’t cause drownings)
• •guarantee independence from $\rho=0$ (nonlinear dependence exists)
• •detect strong nonlinear patterns (U-shapes can yield $\rho \approx 0$)

Why the definition looks the way it does

We want a measure that’s:

1. 1)Positive when both variables tend to be on the same side of their means.
2. 2)Negative when they tend to be on opposite sides.
3. 3)Larger when these deviations are larger.

The product $(X-E[X])(Y-E[Y])$ does exactly that:

• •both positive → product positive
• •one positive, one negative → product negative
• •bigger deviations → bigger magnitude

Averaging (expectation) makes it a stable summary.

Expanding covariance: a very useful formula

Start from the definition:

Expand step-by-step:

So:

This identity is especially handy in calculations.

Covariance with constants and scaling

These properties matter constantly in modeling:

1) Adding constants doesn’t change covariance:

Reason: centering removes constants.

2) Scaling scales covariance:

So if you convert meters to centimeters (×100), covariance changes by ×100 on that variable’s side.

Relationship to variance

Variance is just covariance with itself:

That makes covariance a true generalization of variance.

Independence implies zero covariance (but not conversely)

If $X$ and $Y$ are independent, then $E[XY] = E[X]E[Y]$, so:

But the reverse is not generally true: $\operatorname{Cov}(X,Y)=0$ does not guarantee independence. (We’ll build a concrete counterexample later.)

Sample covariance: estimating from data

In practice, you usually have samples $(xᵢ, yᵢ)$ for $i=1..n$.

Define sample means:

• •$\bar{x} = \frac{1}{n}\sum_{i=1}^n xᵢ$
• •$\bar{y} = \frac{1}{n}\sum_{i=1}^n yᵢ$

A common estimator (unbiased for i.i.d. samples) is:

You may also see $\frac{1}{n}$ instead of $\frac{1}{n-1}$; that version is the maximum-likelihood estimator under a normal model but is biased in finite samples.

Interpreting magnitude: why covariance is hard to compare

Suppose:

• •height in meters vs weight in kg
• •height in centimeters vs weight in kg

The relationship is identical, but scaling height by 100 scales covariance by 100. That’s why correlation is so widely used: it removes units.

Why standardize?

Covariance answers “do they move together?”, but not “how strong is that co-movement relative to each variable’s natural scale?”

Correlation fixes this by dividing by $\sigma_X\sigma_Y$.

Because $\sigma_X$ has units of $X$ and $\sigma_Y$ has units of $Y$, the units cancel.

Correlation is bounded between −1 and 1

This isn’t just a convention—it’s a theorem. One route uses Cauchy–Schwarz.

Let $X_c = X - E[X]$ and $Y_c = Y - E[Y]$.

Apply Cauchy–Schwarz to random variables:

But $E[X_c^2]=\operatorname{Var}(X)=\sigma_X^2$ and similarly for $Y$.

So:

Divide both sides by $\sigma_X\sigma_Y$ (assuming nonzero):

When do we get ±1?

Equality in Cauchy–Schwarz occurs when one variable is an exact scalar multiple of the other (almost surely) after centering.

That means:

• •$\rho=1$ iff $Y - E[Y] = a(X - E[X])$ with $a>0$ a.s.
• •$\rho=-1$ iff same with $a<0$

Equivalently, $Y = aX + b$ almost surely for some constants $a, b$.

A visual: correlation as “angle” between centered coordinates (sample view)

For a dataset, define centered vectors:

• •x = $(x₁-\bar{x},\dots,x_n-\bar{x})$
• •y = $(y₁-\bar{y},\dots,y_n-\bar{y})$

Then the sample correlation is:

That is exactly the cosine of the angle between x and y in $\mathbb{R}^n$.

• •If x and y point in the same direction → $r\approx 1$
• •Orthogonal → $r\approx 0$
• •Opposite directions → $r\approx -1$

Inline “plot” of the vector-angle intuition:

           y (centered)
           ^
          /|
         / |   cos(θ) = r
        /θ |
       +---+----------> x (centered)

This is a powerful mental model: correlation is alignment between patterns of deviation-from-mean.

Sample correlation

Using the sample covariance $s_{xy}$ and sample standard deviations $s_x, s_y$:

Correlation is not robust to outliers

Because it depends on products of deviations, a single extreme point can dominate.

Practical note:

• •If you suspect outliers or heavy tails, consider rank-based alternatives like Spearman’s $\rho$ or Kendall’s $\tau$ (different nodes/topics).

1) The “ice cream and drownings” lesson: correlation ≠ causation

A strong correlation can arise from:

• •a direct causal link ($X \to Y$)
• •reverse causation ($Y \to X$)
• •a third variable $Z$ driving both ($Z \to X$ and $Z \to Y$)
• •coincidence / selection bias / measurement artifacts

In the ice-cream example, temperature (a confounder) increases both ice cream consumption and swimming, which increases drowning risk.

Covariance/correlation quantify association; causal claims require additional design/assumptions.

2) Zero covariance: what it means, and the classic trap

$\operatorname{Cov}(X,Y)=0$ means the linear term in their relationship is absent in a precise sense:

• •If you try to predict $Y$ using a linear function of $X$, the best linear predictor doesn’t gain anything from $X$ (in a mean-square sense) beyond the intercept.

But $X$ can still strongly determine $Y$ nonlinearly.

We’ll show an explicit example in the worked examples: $X \sim \text{Uniform}(-1,1)$ and $Y=X^2$. Then covariance is 0, yet $Y$ is completely determined by $X$.

Special case worth knowing: if $(X,Y)$ are jointly normal, then zero covariance does imply independence. (Not true in general.)

3) Covariance matrix: scaling up to many variables

With $d$ random variables, stack them into a random vector X ∈ $\mathbb{R}^d$.

Define the covariance matrix:

Key facts:

• •Diagonal entries: variances ($\Sigma_{jj}=\operatorname{Var}(X_j)$)
• •Off-diagonals: covariances ($\Sigma_{jk}=\operatorname{Cov}(X_j, X_k)$)
• •$\Sigma$ is symmetric and positive semidefinite

This matrix is the central object in multivariate statistics.

4) Connection to PCA (what this node unlocks)

PCA looks for directions in feature space with maximum variance.

If you have centered data vectors x ∈ $\mathbb{R}^d$, PCA finds unit vectors v maximizing:

The solutions are eigenvectors of the covariance matrix $\Sigma$.

So learning covariance is not just about pairwise relationships—it’s about understanding the geometry of data clouds and the linear structure PCA extracts.

5) A quick “what to use when” table