Embeddings (Dense Representations)

Why we need embeddings (motivation)

Neural networks are built out of operations like matrix multiplication, addition, and nonlinearities. Those operations expect numbers. But many ML inputs are discrete symbols:

• •Words/tokens ("cat", token id 1729)
• •Users/items in recommender systems
• •Nodes in a graph
• •Categorical features (country, device type)

A discrete ID by itself has no meaningful arithmetic: token 5 is not “closer” to token 6 than to token 5000. If we fed the raw integer ID into a model, we’d be smuggling in a fake ordering.

A classic alternative is a one-hot vector eₓ ∈ {0,1}^{|V|}, where |V| is the vocabulary size. This avoids the fake ordering, but it creates a different problem:

• •It’s extremely high-dimensional (|V| can be 50k–500k+).
• •It’s extremely sparse (exactly one 1).
• •Similarity is unhelpful: for x ≠ y, eₓ · eᵧ = 0.

We want a representation where:

1. 1)The model can learn similarity and relationships.
2. 2)The representation is dense and low-dimensional.
3. 3)The representation is trainable as part of the model.

That’s exactly what embeddings provide.

Definition

An embedding is a learned mapping

x ↦ vₓ, where vₓ ∈ ℝᵈ

• •x is a discrete item (a token id, category id, etc.)
• •d is the embedding dimension (e.g., 64, 256, 768, 4096)
• •vₓ is a dense vector of real numbers

We’ll use the key symbol from this node:

• •vₓ : the embedding vector for item x

Embedding table as parameters

In practice, we store embeddings in a matrix (often called an embedding table):

E ∈ ℝ^{|V|×d}

Row x of E (written E[x, :]) is the embedding for item x:

vₓ = E[x, :]

This is why the node description says “embeddings as model parameters”: the rows of E are trainable parameters just like weights in a linear layer.

A key equivalence: lookup = linear algebra

Even though implementations often do a fast “lookup”, it is mathematically equivalent to multiplying by a matrix.

Let eₓ be the one-hot vector for item x (length |V|). Then:

vₓ = eₓᵀ E

Check the shapes:

• •eₓᵀ is 1×|V|
• •E is |V|×d
• •result is 1×d (a row vector, i.e., an embedding)

So an embedding layer is essentially a linear layer with a very special input (one-hot), where the computation reduces to selecting a row.

Dense representation: what “dense” really means

“Dense” means most coordinates are nonzero, and information is distributed across coordinates. This matters because:

• •Similarity can be expressed by overlap across many dimensions.
• •Small changes to parameters can smoothly move vₓ, giving the optimizer (gradient descent) a continuous surface to improve.

Embedding vectors are not “features you interpret one-by-one” in general. They’re coordinates in a learned space.

A mental picture

Think of each item x as a point in d-dimensional space. The model learns where to place those points so that tasks become easy: similar items end up near each other; items that interact strongly get high dot products; offsets can represent relations.

This geometric picture becomes especially important once you learn attention: attention weights are computed from dot products between learned projections of embeddings. So if you can reason about dot products and angles, you can reason about why attention focuses where it does.

Why this mechanic matters

Embeddings are “just parameters,” but they behave differently from typical weight matrices:

• •Only a few rows are touched per batch (sparse updates).
• •Each row corresponds to a discrete item, which might be rare.
• •Regularization and initialization strongly affect stability.

Understanding how embeddings are represented and updated helps you debug training, reason about capacity, and understand common modeling choices.

The embedding layer as a parameter matrix

Let the vocabulary be V with |V| items. The embedding table is

E ∈ ℝ^{|V|×d}

Given an input sequence of token ids (x₁, x₂, …, xₙ), the embedding layer outputs a matrix of vectors:

X =

[ v_{x₁}

v_{x₂}

⋮

v_{xₙ} ] ∈ ℝ^{n×d}

This X is what subsequent layers (like attention) operate on.

One-hot multiplication viewpoint (useful for gradients)

Write the one-hot matrix for the whole sequence as:

H ∈ {0,1}^{n×|V|}

Each row i is e_{xᵢ}ᵀ. Then:

X = H E

This view makes it clear that embeddings are linear in E.

How gradients flow to embeddings

Suppose your loss is L. You compute ∂L/∂X from later layers, then backpropagate to E.

From X = H E, using matrix calculus:

∂L/∂E = Hᵀ (∂L/∂X)

What does this mean intuitively?

• •Hᵀ “routes” gradient rows back to the corresponding embedding rows.
• •Only the rows that appear in the batch get nonzero gradient.

If token x never appears, its row E[x, :] never updates.

Sparse updates and rare tokens

Because updates are sparse:

• •Frequent tokens get many updates → well-trained vectors.
• •Rare tokens get few updates → noisy or undertrained vectors.

This is one reason why tokenization (e.g., subword units) helps: it increases coverage so that even rare words share pieces that appear often.

Initialization and scale

Embeddings are typically initialized with small random values, e.g.

E[x, j] ~ 𝒩(0, σ²)

Why small?

• •If embeddings are too large, early dot products can explode.
• •In attention, dot products are central; large norms can dominate similarity.

Transformers also use scaling (like dividing by √d in attention) partly to control magnitude. But stable embedding scales still matter.

Regularization: keeping embeddings “well-behaved”

Because embedding rows are free to drift, it’s common to apply:

• •Weight decay (L2): add λ‖E‖²_F
• •Norm clipping or max norm constraints: ‖vₓ‖ ≤ c

Why? If some vectors become huge, dot products become dominated by norms rather than directions.

Parameter count and memory

Embedding tables can dominate parameter count.

Parameter count = |V|×d

Example:

• •|V| = 50,000
• •d = 1,024

Then parameters = 50,000×1,024 = 51,200,000 ≈ 51M parameters.

This affects:

• •GPU memory
• •bandwidth (fetching embeddings)
• •training speed

Input embeddings vs output embeddings (tying)

In language models, you often have:

• •An input embedding table E_in ∈ ℝ^{|V|×d}
• •An output weight matrix W_out ∈ ℝ^{|V|×d} used to produce logits: logits = X W_outᵀ

A common trick is weight tying:

W_out = E_in

This reduces parameters and often improves generalization because the model uses a shared notion of token similarity for both reading and writing.

Summary of the mechanics

An embedding layer is a learned matrix where each row is a vector vₓ. Forward pass is indexing; backward pass updates only rows used in the batch. Scale, regularization, and data frequency strongly affect how good those learned vectors become.

Why geometry is the point

Once tokens become vectors, the model can compare them using geometry. This is not a metaphor: attention scores are built from inner products of projected embeddings.

So you want to be fluent with the main geometric tools:

• •Dot product a·b
• •Norm ‖a‖
• •Cosine similarity cos(θ)
• •Euclidean distance ‖a − b‖

Dot product: alignment + magnitude

For a, b ∈ ℝᵈ:

a·b = ∑ⱼ aⱼ bⱼ

It can be decomposed as:

a·b = ‖a‖ ‖b‖ cos(θ)

where θ is the angle between vectors.

So a large dot product can happen because:

1. 1)The vectors point in similar directions (cos(θ) large), and/or
2. 2)One or both vectors have large norms.

This matters because many models treat dot products as “similarity.” But dot products mix direction and scale.

Cosine similarity: pure direction

Cosine similarity removes magnitude:

cos_sim(a, b) = (a·b) / (‖a‖ ‖b‖)

This lives in [-1, 1] (for nonzero vectors) and measures angular closeness.

In practice:

• •Cosine similarity is common for retrieval / nearest neighbors.
• •Attention uses dot products directly (with scaling), so norms can still matter.

Euclidean distance and its relation to dot product

A useful identity connects distance and dot product:

‖a − b‖² = (a − b)·(a − b)

= a·a − 2a·b + b·b

= ‖a‖² − 2a·b + ‖b‖²

If all vectors are normalized to unit norm (‖a‖ = ‖b‖ = 1), then:

‖a − b‖² = 2 − 2(a·b)

So maximizing dot product is equivalent to minimizing distance when norms are controlled.

“Semantic similarity” is task-defined

It’s tempting to say “embeddings encode meaning,” but the meaning is shaped by:

• •The training objective (predict next token, classify sentiment, etc.)
• •The model architecture (attention, MLPs)
• •The data distribution

Two words are “similar” in embedding space if the model benefits from treating them similarly for the task.

In language modeling, similarity often reflects:

• •Similar contexts (“cat” and “dog”)
• •Syntactic roles (“is”, “was”)
• •Topical associations (“Paris”, “France”)

Relationship vectors and offsets (the classic intuition)

Sometimes relations are approximately linear in embedding space. A famous pattern (not a guarantee) is:

v_{king} − v_{man} + v_{woman} ≈ v_{queen}

Interpreting this:

• •The offset v_{king} − v_{man} might represent a “male royalty” relation.
• •Adding v_{woman} shifts toward “female royalty.”

Why can this happen?

If many relationships behave consistently, a linear subspace can represent them efficiently. But remember: this depends on training and is approximate.

Analogies as nearest-neighbor search

To find “queen” from the analogy, you compute:

u = v_{king} − v_{man} + v_{woman}

Then search for the token y whose embedding vᵧ is closest to u (often by cosine similarity):

y = argmaxᵧ cos_sim(u, vᵧ)

This is one place where cosine similarity is preferred: it reduces sensitivity to norm differences.

An important nuance: embedding geometry is not unique

If a downstream layer applies a linear map, then rotating embedding space can preserve behavior.

For example, if you replace embeddings with E′ = E R where R is an orthogonal matrix (RᵀR = I), and adjust the next linear layer accordingly, many computations can remain unchanged.

So individual coordinates rarely have stable “human meanings.” What’s stable is the relative geometry that the network uses.

Comparing similarity measures (table)

Takeaway

Geometric semantics means the model learns a space where “useful relationships” become simple geometric relationships—often high dot products, small angles, or consistent offsets. This geometric viewpoint is not optional: it’s the language attention uses.

Why embeddings are central to attention

Attention mechanisms don’t operate on token ids. They operate on vectors.

In a Transformer, the pipeline starts with embeddings:

1. 1)Token id xᵢ
2. 2)Embedding lookup → v_{xᵢ} ∈ ℝᵈ
3. 3)Add position information (positional embedding or encoding)
4. 4)Apply linear layers to form queries/keys/values

So if embeddings are poorly trained or poorly scaled, attention has bad raw material.

How embeddings feed into Q/K/V projections

Let X ∈ ℝ^{n×d} be the stacked token embeddings.

A single attention head uses learned matrices:

W_Q ∈ ℝ^{d×dₖ}, W_K ∈ ℝ^{d×dₖ}, W_V ∈ ℝ^{d×dᵥ}

Then:

Q = X W_Q

K = X W_K

V = X W_V

The attention score between token i and token j is based on a dot product:

score(i, j) = ( qᵢ · kⱼ ) / √dₖ

and attention weights are softmaxed scores.

Notice the chain:

xᵢ → v_{xᵢ} → qᵢ

xⱼ → v_{xⱼ} → kⱼ

So the embedding geometry is transformed, but it strongly influences how easily the model can learn useful dot products.

Why √dₖ scaling exists (embedding magnitude connection)

If components of q and k have variance σ², the dot product sums dₖ terms, so its variance grows with dₖ. Roughly:

Var(q·k) ∝ dₖ

Dividing by √dₖ keeps the scale of scores more stable as dₖ changes, preventing softmax from saturating too early.

Even with scaling, extreme embedding norms can still cause issues upstream. That’s why scale/initialization/normalization choices around embeddings matter.

Positional embeddings: another embedding table

Tokens alone are insufficient: the sequence order matters.

A common approach is to learn a second embedding table for positions:

P ∈ ℝ^{L×d}

For position i, add:

hᵢ = v_{xᵢ} + pᵢ

Now hᵢ is what the model projects into Q/K/V.

This mirrors the main idea: even “position” is treated as a discrete item that gets a learned dense vector.

Embeddings in other domains

Embeddings are a general pattern:

• •Recommenders: users and items embedded; dot product predicts affinity.
• •Graphs: node embeddings; proximity predicts edges.
• •Multimodal: text embeddings and image embeddings aligned so that matching pairs have high similarity.

The core idea remains: map discrete or complex objects into a shared vector space where geometry supports the task.

Learned similarity vs human similarity

In attention, the model learns to create Q/K spaces where dot products reflect “should attend.” This is a task-specific similarity.

For example:

• •A pronoun embedding may attend to a noun not because the words are similar, but because they are coreferent.
• •A comma may attend to syntactic boundaries.

So embeddings are not just about synonyms—they are about making the downstream operations (especially dot products) predictive.

Practical diagnostics and intuition

When embeddings go wrong, you often see:

• •Slow training: the model struggles to form useful neighborhoods.
• •Instability: exploding norms, attention saturation.
• •Poor generalization for rare tokens.

Interventions often involve:

• •Changing d (capacity)
• •Regularizing norms
• •Better tokenization
• •Weight tying
• •Pretraining (starting from a good embedding space)

Connection forward

Once embeddings are clear, attention becomes less mysterious: it’s “compare vectors, mix values.” You’ll be ready to reason about attention heads as geometric comparators operating on embedded tokens.

Next node:

• •Attention Mechanisms