Cosine Similarity

This node is meant to be foundational, but cosine similarity does assume a few micro-skills. Here’s a compact checklist.

You should recognize these ideas

Two common terminology notes

1. 1)Cosine similarity is the value $\cos\theta$ (range [−1, 1] in general).
2. 2)People often define cosine distance as $1 - \mathrm{cosSim}(\mathbf{a},\mathbf{b})$. This is useful, but it is not guaranteed to be a metric on all vectors because the triangle inequality can fail. (It may behave metrically only under additional constraints, e.g., certain normalized nonnegative settings.)

If those boxes feel unfamiliar, you can still proceed—just revisit them when the formulas appear.

Why we need a “direction-only” similarity

Suppose you have two vectors representing items:

• •in search: document embedding vs query embedding
• •in recommendation: user embedding vs item embedding
• •in NLP: token embeddings interacting inside attention

Often, the direction encodes the “type” or “meaning,” while the length might reflect confidence, frequency, or just the model’s internal scaling.

If we used only the dot product $\mathbf{a}\cdot\mathbf{b}$ to score similarity, then simply making vectors longer (larger magnitude) would inflate the score—even if the direction stayed the same. That can be undesirable when you want comparisons to be about alignment.

Cosine similarity fixes this by normalizing out the magnitudes.

Definition

For two nonzero vectors a and b in ℝⁿ, cosine similarity is

Geometric meaning: “cosine of the angle”

There is a key identity connecting dot product and angle:

where $\theta$ is the angle between a and b (in the usual Euclidean geometry). Rearranging gives

So cosine similarity literally is the cosine of that angle.

Interpreting the value

Because cosine ranges between −1 and 1:

• •1: vectors point the same direction (θ = 0°)
• •0: vectors are orthogonal / perpendicular (θ = 90°)
• •−1: vectors point in opposite directions (θ = 180°)

In many embedding systems (especially after certain training setups), values are often mostly positive, but mathematically the full range [−1, 1] is possible.

Important constraint: nonzero vectors

The formula divides by $\|\mathbf{a}\|\,\|\mathbf{b}\|$. If either vector is 0, then cosine similarity is undefined.

In practice, systems either:

• •ensure embeddings are never zero,
• •add a tiny ε to the denominator, or
• •define a special-case behavior (but this is application-specific).

The dot product mixes direction and magnitude

The dot product is

It increases when:

1) components match in sign and are large in magnitude, and

2) the vectors point in similar directions.

But here’s the catch: if you scale a by a constant $c$, then

So dot product is not scale-invariant.

A concrete example: same direction, inflated score

Let

• •a = (1, 1)
• •b = (2, 2)
• •c = (100, 100)

These all point in the same direction (45° line). Dot products:

• •a·b = 1·2 + 1·2 = 4
• •a·c = 1·100 + 1·100 = 200

The second looks “more similar” by dot product, but b and c are equally aligned with a—they just have different lengths.

Normalization removes the scale

Cosine similarity divides by both lengths:

Now see what happens if we scale a by $c>0$:

\nShow your work (scale invariance)

Let a' = ca.

1) Dot product scales:

2) Norm scales:

3) Plug into cosine similarity:

So cosine similarity does not change when you scale one vector by a positive constant.

Unit vectors make the idea even clearer

Define normalized (unit-length) vectors:

Then

This is a powerful mental model:

• •cosine similarity = dot product after both vectors are projected onto the unit sphere.
• •all comparisons become “angle comparisons” rather than “length comparisons.”

Range and the Cauchy–Schwarz inequality

A key guarantee is that cosine similarity is always between −1 and 1.

Cauchy–Schwarz says:

Divide both sides by $\|\mathbf{a}\|\,\|\mathbf{b}\|$ (nonzero assumption):

So

This boundedness is one reason cosine similarity is numerically and conceptually convenient.

Cosine similarity is fundamentally about angle

Because $\mathrm{cosSim}(\mathbf{a},\mathbf{b}) = \cos\theta$, it inherits the cosine curve’s behavior:

• •Small angles (θ near 0): cosine near 1 → high similarity
• •Medium angles (θ near 90°): cosine near 0 → “unrelated directions”
• •Large angles (θ near 180°): cosine near −1 → opposite directions

This is slightly different from many intuitive “distance” notions.

Negative cosine similarity: when opposite directions matter

In some applications, negative similarity has a clear meaning:

• •In sentiment-like axes, one direction might correspond to “positive,” opposite to “negative.”
• •In some factor models, opposite direction can encode opposing preferences.

In other applications (e.g., some retrieval systems), negative values might just be treated as “not similar” and thresholded away.

Relation to Euclidean distance (when vectors are normalized)

If both vectors are normalized to unit length, then cosine similarity and Euclidean distance are tightly connected.

Let $\hat{\mathbf{a}}$ and $\hat{\mathbf{b}}$ be unit vectors. Consider squared Euclidean distance:

\nShow your work

Expand:

Since both are unit length:

So

Rearrange:

So on the unit sphere, cosine similarity is basically a monotonic transformation of Euclidean distance.

Cosine distance is common but not a guaranteed metric

A frequently used “distance-like” quantity is

This has nice properties (0 when vectors match in direction, bigger when they differ), but it is not always a metric on ℝⁿ. In particular, the triangle inequality may fail.

Why that matters:

• •Some algorithms (certain clustering/indexing schemes) assume a true metric.
• •If you plug in a non-metric distance, you can get subtle correctness/performance issues.

Practical takeaway: it’s fine to use $1-\cos$ as a heuristic distance for ranking and optimization, but don’t automatically assume metric guarantees unless you’ve checked the conditions for your specific setting.

Implementation note: numerical stability

When vectors are very small or nearly zero, the denominator $\|\mathbf{a}\|\,\|\mathbf{b}\|$ can be tiny.

Common fix:

or add ε inside the product. The exact choice depends on your numerical environment and expectations about zero vectors.

1) Comparing embeddings (semantic similarity)

Embeddings map discrete objects (words, items, images) to vectors. A core assumption is:

• •direction corresponds to “meaning/features”
• •closeness in angle corresponds to similarity

Cosine similarity is a natural fit because it ignores overall magnitude. This is especially helpful when vector norms vary due to:

• •frequency effects (common words can have different scales)
• •training dynamics
• •model architecture (some layers output vectors with varying norms)

Typical retrieval pipeline

1. 1)Compute embedding for query q and for each candidate item xᵢ.
2. 2)Score each candidate with $\mathrm{cosSim}(\mathbf{q},\mathbf{x}_i)$.
3. 3)Return top-k.

Often, systems normalize all embeddings once (store unit vectors), making scoring just a dot product.

2) Cosine similarity as an attention score

In attention mechanisms, we produce:

• •queries q
• •keys k
• •values v

A general attention score between a query and key can be any similarity function. One simple option is cosine similarity:

Then attention weights are typically:

In modern transformers, the most common is scaled dot-product attention:

Why not always cosine?

• •Cosine similarity forces magnitude invariance; sometimes magnitude contains useful information.
• •Scaled dot product is cheaper if you already have vectors and don’t want norms.
• •With layer normalization and training, dot-product attention can behave stably.

That said, cosine attention variants exist and can be helpful in some regimes.

3) Practical tradeoffs: dot product vs cosine similarity

4) A workflow pattern: normalize once, then dot

Because

a very common engineering trick is:

1) store $\hat{\mathbf{x}} = \mathbf{x}/\|\mathbf{x}\|$ for every embedding

2) compute similarity as a dot product

This can speed up retrieval (especially with vector databases / ANN indices) because dot products are highly optimized.