Vector Embeddings

Why we need them (motivation)

Discrete items—like words, tokens, product IDs, user IDs, categorical features, or graph nodes—don’t naturally live in a space where “closeness” or “similarity” is meaningful.

• •If we label words by integers (cat=17, dog=53), then 53 − 17 = 36 is meaningless.
• •If we use one-hot vectors, the representation is huge and every distinct item is orthogonal to every other (cosine similarity 0), so geometry cannot express “cat is closer to dog than to spaceship.”

Embeddings solve this by giving each item a dense vector eₓ ∈ ℝᵈ. Once items are vectors, we can compare them with dot products, cosine similarity, and distances. That makes “interaction” between items (in attention, retrieval, classification, etc.) a simple geometric computation.

Definition

An embedding is a function (often learned) that maps a discrete item x to a continuous vector:

• •x ∈ V (a vocabulary / set of IDs)
• •eₓ ∈ ℝᵈ (a d-dimensional real vector)

We write:

• •eₓ = Emb(x)

In many neural networks, Emb(·) is implemented as a lookup table (a matrix) with one vector per item.

Intuition: “meaning as location”

The core intuition is not that embeddings store dictionary definitions. Instead:

• •Items that behave similarly in the model’s task get pushed to similar vectors.
• •“Similarity” becomes “close direction” (cosine similarity) or “close point” (distance).

For language, distributional learning yields the classic idea: “You know a word by the company it keeps.” If two words appear in similar contexts, a training objective will encourage their embeddings to become similar.

Dense vs sparse representations

A one-hot vector x ∈ {0,1}^|V| is sparse: exactly one 1 and the rest 0. An embedding eₓ ∈ ℝᵈ is dense: typically all d components are nonzero.

A useful comparison:

Embeddings are “compact but expressive.” They trade fixed semantics (one-hot) for learnable semantics (geometry).

The key symbol

We’ll use eₓ to denote the embedding vector for item x:

• •eₓ ∈ ℝᵈ

And we’ll often compare embeddings using cosine similarity (prerequisite):

cos(a, b) = (a · b) / (‖a‖ ‖b‖)

Cosine similarity focuses on direction, which is often what matters in learned representation spaces.

What embeddings are not

1. 1)Not uniquely defined: Many embedding spaces are equivalent up to rotation/reflection, because objectives depend on dot products and distances.
2. 2)Not guaranteed “human semantic”: They capture what helps the training loss, which may include biases or spurious correlations.
3. 3)Not always global coordinates: Some models care about relative comparisons (angles/dots) more than absolute axes.

Embeddings are a learned coordinate system designed to make downstream computation easy.

Why a lookup table works

When x is a discrete ID, the simplest parameterization is: assign a trainable vector to each ID. Collect these vectors into a matrix E.

Let:

• •Vocabulary size: |V|
• •Embedding dimension: d
• •Embedding matrix: E ∈ ℝ^{|V|×d}

Row i of E is the embedding for token i:

• •eᵢ = E[i]

This is literally a learned table.

From one-hot to embedding: a clean algebraic view

If x is a one-hot vector representing item i, then the embedding lookup is equivalent to matrix multiplication:

• •x ∈ {0,1}^{|V|}
• •E ∈ ℝ^{|V|×d}

Compute:

e = xᵀ E

Because x has a single 1 at index i, xᵀ E selects the i-th row.

This equivalence is useful for understanding gradients: the model updates the row(s) corresponding to the IDs it sees.

How embeddings get trained (gradient intuition)

Embeddings are learned because they participate in a loss. The loss might come from:

• •Next-token prediction (language modeling)
• •Masked token prediction
• •Contrastive objectives (push positives together, negatives apart)
• •Classification/regression (embedding feeds a predictor)

Regardless of the objective, the embedding vectors are parameters. During backprop, the gradient ∂L/∂eₓ updates eₓ.

A simple mental model:

• •If the model wants x to behave more like y, it nudges eₓ toward eᵧ (in direction).
• •If it wants them distinct, it nudges them apart.

A concrete training objective: softmax classifier from embeddings

Consider a very common pattern: predict a label y from an embedding h (which could be eₓ or a contextual vector). Use a linear layer and softmax:

• •logits: z = W h + b
• •probabilities: p = softmax(z)

Loss for true class c:

L = −log p_c

Even if you don’t memorize softmax gradients, the key is:

• •h is pushed to increase the logit of the correct class
• •that pressure flows backward into embeddings if h depends on them

So embeddings become whatever vectors make the rest of the network succeed.

Embedding dimension d: capacity vs generalization

Why not make d enormous?

• •Larger d increases capacity: more ways to place items distinctly.
• •But it increases parameters: |V|·d, memory, and risk of overfitting (especially for rare items).

A useful rule of thumb: pick d based on data scale, vocabulary size, and task complexity. In transformers, d often matches the model width so embeddings can be added to positional encodings and fed into attention.

Weight tying: input embeddings and output embeddings

In language models, there are often two related matrices:

• •Input embeddings: map token IDs → vectors
• •Output weights: map hidden state → vocabulary logits

Sometimes these are tied (shared): output weight matrix W is set to Eᵀ.

Why tie?

• •Reduces parameters
• •Encourages a consistent geometry: tokens used as inputs live in the same space as tokens being predicted

Initialization and scale

Embeddings are usually initialized randomly with small variance. Scale matters because dot products and norms affect attention scores and softmax logits.

In attention, if dot products get too large, softmax can saturate (become too peaky). That’s one reason scaled dot-product attention uses 1/√d.

Even before attention, stable embedding scales help optimization.

Special tokens and feature embeddings

Embeddings aren’t just for words:

• •[CLS], [SEP], [MASK] tokens
• •categorical features (country, device type)
• •discretized numeric bins
• •IDs (user, item)

In all cases, the embedding is a learned vector that becomes a “handle” for the model to condition on.

Summary of this mechanic

• •Embeddings often live in a matrix E ∈ ℝ^{|V|×d}
• •Lookup is row selection (or one-hot × matrix)
• •Training nudges vectors to support a task objective
• •d controls capacity and compute/memory

This gives us the basic object: eₓ. Next we’ll focus on the geometry of embeddings—what dot products and angles mean, and why that geometry becomes the substrate for attention.

Why geometry matters

Once items are vectors, the model can compute interactions with fast linear algebra. The most common interaction is a dot product:

• •a · b = ∑ᵢ aᵢ bᵢ

Dot products are cheap, differentiable, and deeply connected to angles and lengths. This is why embeddings pair naturally with attention.

But: dot products depend on both direction and magnitude. Cosine similarity removes magnitude to focus on direction:

cos(a, b) = (a · b) / (‖a‖ ‖b‖)

So you should keep two pictures in mind:

1. 1)Directional similarity (angle) → cosine similarity
2. 2)Positional proximity (distance) → Euclidean distance

Angle vs distance

For a, b ∈ ℝᵈ:

• •Euclidean distance: ‖a − b‖
• •Squared distance: ‖a − b‖²

Expand squared distance:

‖a − b‖²

= (a − b) · (a − b)

= a·a − 2a·b + b·b

= ‖a‖² − 2(a·b) + ‖b‖²

This shows a key link:

• •If norms are similar (‖a‖ ≈ ‖b‖), then minimizing distance is roughly equivalent to maximizing dot product.

If we L2-normalize embeddings (force ‖eₓ‖ = 1), then:

‖a − b‖² = 2 − 2(a·b)

and since a·b = cos(a, b) for unit vectors:

‖a − b‖² = 2 − 2 cos(a, b)

So for normalized embeddings:

• •maximizing cosine similarity ⇔ minimizing Euclidean distance

This is why many retrieval systems store normalized embeddings.

What makes embeddings “semantic” (or functional)

Embeddings encode whatever similarity the training process rewards.

• •In word2vec-style objectives, words sharing contexts end up near each other.
• •In recommender systems, users/items that co-occur in interactions become aligned.
• •In supervised tasks, categories that share predictive patterns can cluster.

The geometry becomes a compressed record of these pressures.

Similarity is task-relative

Two important consequences:

1. 1)If you train embeddings for sentiment classification, “good” and “great” may cluster, but also “awful” and “terrible.” That’s still semantic—but shaped by sentiment.
2. 2)If you train embeddings for code completion, “for” and “while” may cluster functionally.

So “semantic similarity” is better read as useful similarity under the objective.

The dot product as “compatibility”

In many neural modules, the dot product between embeddings stands for compatibility.

• •High q·k → query and key match
• •High u·v → user and item match

This is a learned notion of compatibility because q, k, u, v come from learned embeddings or learned projections of embeddings.

Anisotropy and frequency effects

Real embedding spaces often have quirks:

• •Frequency dominance: common tokens can become “central,” showing high similarity with many tokens.
• •Anisotropy: vectors may cluster in a narrow cone rather than filling the sphere.

These effects can harm retrieval or similarity search because cosine similarities become less discriminative.

Mitigations include:

• •mean-centering embeddings
• •whitening / PCA postprocessing
• •training objectives that encourage uniformity
• •normalization and temperature scaling

Regularization and constraints

Sometimes we add constraints to shape geometry:

• •L2 regularization encourages small norms
• •normalization encourages unit norms
• •contrastive losses explicitly control angles via margins/temperatures

Each choice changes how dot products translate into probabilities.

Embeddings as points, but also as basis coordinates

Another way to view embedding components:

• •Each dimension can be seen as a latent feature axis.
• •But axes are not individually interpretable in general; meaning is distributed across dimensions.

You’ll occasionally find interpretable directions (e.g., sentiment), but that’s not guaranteed.

Summary of this mechanic

• •Similarity usually means cosine similarity or dot product
• •For unit vectors, cosine similarity and Euclidean distance are tightly linked
• •Geometry is shaped by the training objective (functional similarity)
• •Dot products act as compatibility scores, setting up attention and retrieval

Why attention needs embeddings

Attention mechanisms operate on vectors. If your input is discrete tokens, you must first map them to vectors—embeddings.

In a transformer, a typical pipeline is:

1. 1)Tokenize text → token IDs x₁, …, xₙ
2. 2)Lookup embeddings → e₁, …, eₙ where eᵢ = e_{xᵢ}
3. 3)Add positional information → hᵢ⁽⁰⁾ = eᵢ + pᵢ
4. 4)Apply attention layers to compute contextual vectors

So embeddings are the “raw material” that attention will mix.

From embeddings to queries/keys/values

Self-attention doesn’t usually compare raw embeddings directly. It projects them:

• •qᵢ = W_Q hᵢ
• •kᵢ = W_K hᵢ
• •vᵢ = W_V hᵢ

Then attention scores use dot products:

score(i, j) = (qᵢ · kⱼ) / √d_k

and weights are softmax over j.

Even though q, k, v are projected, their source is the embedding space. The structure of embeddings strongly influences what the model can learn efficiently:

• •If related tokens begin in reasonable positions, attention can learn useful patterns with fewer updates.
• •If embeddings are poorly trained, attention must compensate by learning more complex projections.

Embeddings beyond tokens

Transformers also embed:

• •positions (positional embeddings)
• •segments (token-type embeddings)
• •patches in vision transformers
• •features in tabular transformers

The concept is the same: a discrete unit becomes a vector so attention can compare and combine units.

Retrieval and nearest neighbors

Embeddings are also used for retrieval:

• •Encode a query to q
• •Encode items/documents to dᵢ
• •Retrieve top-k by cosine similarity or dot product

This is the backbone of semantic search and RAG systems.

If embeddings are normalized, ranking by cosine similarity is equivalent to ranking by dot product.

Recommendation systems

User and item embeddings are classic:

• •u = embedding(user_id)
• •v = embedding(item_id)

A simple predictor is a dot product:

ŷ = u · v

If user u likes items similar to v, the training objective increases u·v, bringing vectors into alignment.

Practical considerations

Memory: embedding tables can dominate parameter count when |V| is large.

• •Parameters: |V|·d
• •Example: |V| = 200k, d = 1024 → ~205 million floats

Common solutions:

• •smaller d
• •vocabulary pruning
• •subword tokenization (reduces |V|)
• •adaptive embeddings / factorization
• •quantization for serving

OOV and rare items:

• •Use subwords (BPE, WordPiece)
• •Use hashing tricks for categorical features
• •Use shared embeddings for unknown/rare buckets

Fine-tuning vs freezing:

• •Fine-tuning changes embeddings, aligning them to a new task
• •Freezing keeps geometry stable, relying on higher layers

Connection forward: why this unlocks Attention Mechanisms

Attention is fundamentally about similarity-weighted mixing. Similarity is computed by dot products between vectors.

Embeddings provide:

• •a vector per discrete unit
• •an initial geometry where dot products/angles encode useful relationships

Once you have eₓ, attention can form q, k, v and compute relevance.

This is why “Vector Embeddings” is a prerequisite: without vectorization, there is no meaningful similarity computation to drive attention.

Next node: Attention Mechanisms.