Projections

Why projections matter (motivation)

In linear algebra, a subspace S is a “legal set of directions.” Often, the vector you have—call it v—doesn’t lie inside S. That mismatch happens constantly:

• •A measurement b doesn’t lie exactly in the column space of a model matrix A.
• •A noisy signal doesn’t lie exactly in the subspace spanned by your chosen basis functions.
• •A complicated vector is approximated using a lower-dimensional subspace for compression.

In all these cases, you want a principled way to replace v with a nearby vector u that does lie in S. The key idea is: choose the closest one.

Definition (orthogonal projection)

Let S ⊂ ℝⁿ be a subspace, and let v ∈ ℝⁿ. The orthogonal projection of v onto S, written proj_S(v), is the unique vector u ∈ S that minimizes the distance to v:

• •u = proj_S(v) is the unique minimizer of

minimize over u ∈ S: ‖v − u‖.

This is a geometric “drop a perpendicular” idea generalized from 2D/3D to ℝⁿ.

The orthogonality condition (what makes it “orthogonal”)

The minimizer isn’t just any closest point. It satisfies a very specific condition:

Let p = proj_S(v). Define the residual (error) r = v − p.

Then:

• •r ⟂ S

Meaning r is orthogonal to every vector in S:

• •∀ s ∈ S, sᵀ(v − p) = 0.

This is the characterization that makes projections computable.

Uniqueness (why there is exactly one best point)

Because S is a subspace (hence closed and convex) and the objective ‖v − u‖² is strictly convex in u, there is exactly one minimizer.

Geometrically: the set of points in S is “flat,” and the distance function has a single lowest point when you look from v toward that flat set.

Decomposing a vector into parallel + perpendicular parts

Orthogonal projection gives a clean decomposition:

• •v = p + r
• •p ∈ S
• •r ∈ S⊥
• •p ⟂ r

So projection is not merely “closest point.” It is a structured split into a component inside the subspace and a component orthogonal to it.

A quick sanity check: 1D subspace (a line)

If S is the span of a nonzero vector a, then the projection is the familiar formula:

• •proj_S(v) = ((aᵀv) / (aᵀa)) a.

If a is unit length (‖a‖ = 1), this simplifies to:

• •proj_S(v) = (aᵀv) a.

The general subspace case is this idea repeated, but with multiple basis directions at once.

Why we need a computable rule

The definition “pick the closest u ∈ S” is conceptually perfect, but it doesn’t yet tell you how to compute u. The trick is to translate “closest” into an equation.

Instead of minimizing ‖v − u‖ directly, minimize the squared norm (same minimizer, easier algebra):

• •minimize over u ∈ S: f(u) = ‖v − u‖².

The key idea: the residual is orthogonal to the subspace

Let p ∈ S be the minimizer. Consider any direction s ∈ S (a feasible direction to move within the subspace). If you nudge p inside the subspace by ts, the distance to v cannot decrease at t = 0 (otherwise p wasn’t minimal).

Compute the derivative:

f(p + ts) = ‖v − (p + ts)‖²

= ‖(v − p) − ts‖²

= ((v − p) − ts)ᵀ((v − p) − ts)

= ‖v − p‖² − 2t sᵀ(v − p) + t²‖s‖².

At a minimum, derivative at t = 0 is 0:

d/dt f(p + ts) |_{t=0} = −2 sᵀ(v − p) = 0.

So:

• •sᵀ(v − p) = 0 for all s ∈ S,

which is exactly:

• •v − p ∈ S⊥.

This is the most important practical fact: the residual is orthogonal to the model subspace.

Using a basis: turning “orthogonal to S” into linear equations

Suppose S is spanned by k linearly independent vectors a₁, …, aₖ. Put them into a matrix A ∈ ℝⁿˣᵏ as columns:

• •A = [a₁ … aₖ]
• •S = Col(A).

Any vector in S looks like Ax for some x ∈ ℝᵏ.

We want p = Ax that makes r = v − Ax orthogonal to S.

Being orthogonal to S is equivalent to being orthogonal to every column of A:

• •aᵢᵀ(v − Ax) = 0 for i = 1..k.

Stack these k equations:

• •Aᵀ(v − Ax) = 0

⇒ AᵀAx = Aᵀv.

These are the normal equations in the language of least squares (we’ll return to that). For projection, they are simply the orthogonality conditions written in matrix form.

Orthonormal basis special case (fast mental model)

If the columns of Q ∈ ℝⁿˣᵏ are orthonormal (QᵀQ = I), then projection is especially simple.

Let S = Col(Q). We want p ∈ S of the form p = Qc.

Use the orthogonality condition:

Qᵀ(v − Qc) = 0

⇒ Qᵀv − QᵀQc = 0

⇒ Qᵀv − Ic = 0

⇒ c = Qᵀv.

So:

• •proj_S(v) = Q(Qᵀv).

Interpretation:

1) Compute coordinates c = Qᵀv (dot products against basis vectors).

2) Rebuild the vector in the subspace: Qc.

This is the cleanest picture of what projection does.

Pythagorean identity (what you gain from orthogonality)

With v = p + r and p ⟂ r:

‖v‖² = ‖p‖² + ‖r‖².

This is often used to reason about error. Projection chooses p to make ‖r‖ as small as possible, so it also makes ‖p‖ as large as possible among all vectors in S that could approximate v.

Why introduce a projection matrix?

If you will project many different vectors onto the same subspace S, you want a reusable operator. Projections are not just geometric constructions; they are linear transformations.

For a fixed subspace S ⊂ ℝⁿ, the orthogonal projection is a function P: ℝⁿ → ℝⁿ such that:

• •P(v) = proj_S(v).

When written in standard coordinates, this function is multiplication by an n×n matrix, also called P.

Column-space projection: P = A(AᵀA)⁻¹Aᵀ

Assume A ∈ ℝⁿˣᵏ has full column rank (its columns are independent). Then AᵀA is invertible.

Given b ∈ ℝⁿ, the projection of b onto Col(A) is p = Ax̂ where x̂ solves the normal equations:

AᵀAx̂ = Aᵀb

⇒ x̂ = (AᵀA)⁻¹Aᵀb.

Plug back into p:

p = Ax̂

= A((AᵀA)⁻¹Aᵀb)

= (A(AᵀA)⁻¹Aᵀ)b.

So the projection matrix is:

• •P = A(AᵀA)⁻¹Aᵀ
• •proj_Col(A)(b) = Pb.

What P does (and why it’s special)

For an orthogonal projection matrix P onto S:

1) Idempotent: P² = P

Apply the projection twice: once you’re in the subspace, projecting again does nothing.

2) Symmetric: Pᵀ = P

This encodes “orthogonal” in matrix form.

3) Range and null space:

• •Range(P) = S
• •Null(P) = S⊥

So P keeps the S component and kills the orthogonal component.

Verifying idempotence (show the work)

Let P = A(AᵀA)⁻¹Aᵀ. Then:

P² = [A(AᵀA)⁻¹Aᵀ][A(AᵀA)⁻¹Aᵀ]

= A(AᵀA)⁻¹(AᵀA)(AᵀA)⁻¹Aᵀ

= A(I)(AᵀA)⁻¹Aᵀ

= A(AᵀA)⁻¹Aᵀ

= P.

The step (AᵀA)⁻¹(AᵀA) = I works because A has full column rank.

Verifying symmetry (show the work)

Pᵀ = [A(AᵀA)⁻¹Aᵀ]ᵀ

= (Aᵀ)ᵀ[(AᵀA)⁻¹]ᵀAᵀ

= A[(AᵀA)⁻¹]Aᵀ

= P,

because AᵀA is symmetric, so (AᵀA)⁻¹ is also symmetric.

Orthonormal columns shortcut: P = QQᵀ

If Q has orthonormal columns (QᵀQ = I), then:

P = Q(QᵀQ)⁻¹Qᵀ = QIQᵀ = QQᵀ.

This is why orthonormal bases are so valuable: they make projection formulas simpler and numerically more stable.

Eigenvalues intuition (optional but useful)

From P² = P, any eigenvalue λ satisfies λ² = λ, so λ ∈ {0, 1}.

• •Eigenvalue 1 directions are vectors in S (they stay unchanged).
• •Eigenvalue 0 directions are vectors in S⊥ (they get projected to 0).

So P is literally a “keep these directions, erase those directions” operator.

Why least squares is projection in disguise

Consider a system Ax = b. If b ∈ Col(A), there is an exact solution. But if b is not in Col(A), there is no x that makes Ax equal to b.

Least squares replaces “solve exactly” with “get as close as possible”:

• •minimize over x: ‖Ax − b‖.

This is the same geometry as projection:

• •The set {Ax : x ∈ ℝᵏ} is exactly Col(A).
• •You are choosing the closest point in Col(A) to b.

So the least-squares fitted vector Ax̂ is:

• •Ax̂ = proj_Col(A)(b) = Pb.

Deriving the normal equations (show the work)

Minimize the squared error:

f(x) = ‖Ax − b‖²

= (Ax − b)ᵀ(Ax − b).

Expand:

f(x) = xᵀAᵀAx − 2xᵀAᵀb + bᵀb.

Take gradient and set to zero:

∇f(x) = 2AᵀAx − 2Aᵀb = 0

⇒ AᵀAx̂ = Aᵀb.

These are the normal equations.

Geometric meaning: the residual r = b − Ax̂ is orthogonal to Col(A):

Aᵀ(b − Ax̂) = 0.

That is exactly the orthogonality condition for projection.

When does the formula (AᵀA)⁻¹Aᵀ work?

If A has full column rank, AᵀA is invertible and:

• •x̂ = (AᵀA)⁻¹Aᵀb
• •Ax̂ = A(AᵀA)⁻¹Aᵀb = Pb.

If A does not have full column rank, there are infinitely many least-squares minimizers; then one typically uses the pseudoinverse A⁺ (outside this node’s core scope, but good to know).

Residual properties you can rely on

Let p = Ax̂ and r = b − p.

• •p ∈ Col(A)
• •r ⟂ Col(A)
• •b = p + r
• •‖b‖² = ‖p‖² + ‖r‖² (Pythagorean)

This gives a stable mental model: least squares chooses the decomposition of b into “explained by the model subspace” plus “leftover orthogonal noise.”

Connection to linear regression

In linear regression, A is the design matrix, x is the parameter vector, and b (often written y) is the target output. The fitted predictions Ax̂ are the projection of y onto Col(A).

That single sentence unifies:

• •why residuals are orthogonal to features,
• •why the normal equations appear,
• •why adding more columns to A can only reduce training error (you enlarge the subspace, so the closest point can only get closer).