Dot Product

Why this concept exists (motivation)

When you have two vectors a and b, you often want a single number that answers: “How similar are their directions?”

• •If a and b point the same way, you want a large positive number.
• •If they are perpendicular, you want 0.
• •If they point opposite ways, you want a negative number.

The dot product is designed to do exactly that, while still behaving nicely with algebra (linearity).

Algebraic definition (component-wise)

For a = (a₁, a₂, …, aₙ) and b = (b₁, b₂, …, bₙ) in ℝⁿ, the dot product is

a · b = ∑ᵢ₌₁ⁿ aᵢ bᵢ

It is a scalar (a single number), not a vector.

To keep the pacing clear, here’s the 2D and 3D special cases:

• •In ℝ²: (a₁, a₂) · (b₁, b₂) = a₁b₁ + a₂b₂
• •In ℝ³: (a₁, a₂, a₃) · (b₁, b₂, b₃) = a₁b₁ + a₂b₂ + a₃b₃

Geometric definition (angle + lengths)

If θ is the angle between nonzero vectors a and b, then

a · b = ‖a‖ ‖b‖ cos θ

This is powerful because it connects an algebraic computation (sum of products) to geometry (angles).

Interpreting the sign and magnitude

From a · b = ‖a‖‖b‖cos θ:

• •If 0 < θ < π/2, then cos θ > 0 ⇒ a · b > 0 (acute angle, aligned).
• •If θ = π/2, then cos θ = 0 ⇒ a · b = 0 (perpendicular).
• •If π/2 < θ < π, then cos θ < 0 ⇒ a · b < 0 (obtuse angle, opposed).

So the dot product is a “direction agreement score,” scaled by how long the vectors are.

A useful mental model: shadow / projection

Think of a · b as: “How much of a lies along b (or vice versa), times ‖b‖.”

More precisely, if b ≠ 0, then

a · b = ‖b‖ · (signed length of the projection of a onto the direction of b)

We’ll make that precise later, but it’s a great intuition for what dot products mean.

How to compute it reliably

Given a, b ∈ ℝⁿ:

1. 1)Pair up corresponding components (aᵢ with bᵢ).
2. 2)Multiply each pair.
3. 3)Sum the products.

Example structure:

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ

Key properties (why we like dot product)

These properties are the reason dot product becomes the default “inner product” in ℝⁿ.

1) Commutative

a · b = b · a

Because multiplication of real numbers is commutative: aᵢbᵢ = bᵢaᵢ.

2) Distributive over addition

a · (b + c) = a · b + a · c

Show it component-wise:

b + c = (b₁ + c₁, …, bₙ + cₙ)

Then

a · (b + c)

= ∑ᵢ aᵢ(bᵢ + cᵢ)

= ∑ᵢ (aᵢbᵢ + aᵢcᵢ)

= ∑ᵢ aᵢbᵢ + ∑ᵢ aᵢcᵢ

= a · b + a · c

This “linearity” is essential for projections and least squares later.

3) Homogeneous with scalars

(ka) · b = k(a · b) and a · (kb) = k(a · b)

Component-wise:

(ka) · b = ∑ᵢ (kaᵢ)bᵢ = k∑ᵢ aᵢbᵢ = k(a · b)

4) Positive definiteness (dot with itself)

a · a ≥ 0, and a · a = 0 ⇔ a = 0

Because

a · a = ∑ᵢ aᵢ²

A sum of squares can’t be negative, and it’s zero only if every aᵢ = 0.

Dot product and length (preview of norms)

This property gives you Euclidean length:

‖a‖ = √(a · a)

Check quickly:

a · a = a₁² + … + aₙ²

So

‖a‖ = √(a₁² + … + aₙ²)

This connection is a big reason the dot product is everywhere: it defines the usual geometry of ℝⁿ.

Table: operations to keep straight

Takeaway: dot product is the main way we turn “two directions” into “one number.”

Why geometry matters here

The algebraic definition ∑ᵢ aᵢbᵢ is easy to compute, but it doesn’t look like an angle measure.

The geometric form

a · b = ‖a‖‖b‖cos θ

explains what the dot product is doing: it compares directions.

Deriving a practical angle formula

If a ≠ 0 and b ≠ 0, you can solve for cos θ:

a · b = ‖a‖‖b‖cos θ

Divide both sides by ‖a‖‖b‖:

cos θ = (a · b) / (‖a‖‖b‖)

Then

θ = arccos( (a · b) / (‖a‖‖b‖) )

This is how you compute angles between vectors in any dimension.

Orthogonality criterion (the dot-product test)

Two vectors are orthogonal (perpendicular) exactly when their dot product is zero:

a ⟂ b ⇔ a · b = 0

Geometric reason:

a · b = ‖a‖‖b‖cos(π/2) = ‖a‖‖b‖·0 = 0

Algebraic usefulness: you don’t need to “draw” vectors to test perpendicularity.

Unit vectors and simplifying the dot product

If you normalize vectors to unit length:

û = a / ‖a‖, v̂ = b / ‖b‖

Then

û · v̂ = cos θ

So for unit vectors, the dot product directly equals cosine similarity.

Cauchy–Schwarz inequality (why cosine stays in range)

A critical fact (you’ll use it constantly later) is:

|a · b| ≤ ‖a‖‖b‖

This implies

-1 ≤ (a · b) / (‖a‖‖b‖) ≤ 1

So arccos is well-defined (up to floating-point errors in practice).

We won’t fully prove Cauchy–Schwarz here, but intuitively: the projection of one vector onto another can’t be longer than the original.

A “projection-flavored” identity to remember

For b ≠ 0, the scalar projection (signed) of a onto b is

comp_{b}(a) = (a · b) / ‖b‖

And the vector projection is

proj_{b}(a) = ((a · b) / (‖b‖²)) b

Notice how the dot product is the engine: it extracts the amount of alignment needed to scale b.

This is the gateway to the node Projections.

1) Similarity in data (ML intuition)

In machine learning and information retrieval, vectors often represent data:

• •A document as a “bag-of-words” vector
• •A user as a preference vector
• •An embedding as a feature vector

The dot product a · b becomes a similarity score: larger means more aligned.

But there’s a catch: dot product depends on length. If one vector has huge magnitude, it can dominate the score.

That motivates cosine similarity:

cos θ = (a · b) / (‖a‖‖b‖)

This measures alignment independent of scale.

2) Projections and least squares (linear algebra intuition)

Suppose you want the “part of a in the direction of b.” The projection formula

proj_{b}(a) = ((a · b) / (‖b‖²)) b

is fundamental for:

• •projecting onto a line or subspace
• •deriving least-squares solutions
• •understanding orthogonality of residuals

In least squares, the key condition is often: residual r is orthogonal to the subspace, meaning r · v = 0 for all basis vectors v in the subspace.

3) Norms and distances

Because ‖a‖ = √(a · a), dot product defines Euclidean geometry.

A classic identity connects dot products and distances:

‖a − b‖² = (a − b) · (a − b)

Expand:

(a − b) · (a − b)

= a · a − a · b − b · a + b · b

= ‖a‖² − 2(a · b) + ‖b‖²

This shows dot products are enough to compute squared distances.

4) Kernel methods: dot products in disguise

Kernel methods (like SVMs with kernels) revolve around computing

ϕ(x) · ϕ(z)

without explicitly forming the feature map ϕ.

A kernel function K(x, z) is essentially a dot product in some (possibly huge) feature space:

K(x, z) = ϕ(x) · ϕ(z)

So understanding dot products deeply is a prerequisite for the “kernel trick” intuition: learning with geometry in feature space while doing computations in input space.

Where this node points next

• •Orthogonality is the “dot product equals zero” worldview.
• •Norms use √(a · a) to define length.
• •Projections use (a · b) / (‖b‖²) as the scaling factor.
• •Kernel methods generalize dot products to nonlinear similarity measures.