Positional Encoding

Why do we need it?

Many sequence models (like RNNs) process tokens in order, so time is baked into the computation: token 7 can only be seen after token 6. Transformers, in contrast, are designed to process a sequence in parallel. Self-attention compares tokens to each other without inherently knowing which token came earlier.

If you hand a Transformer the embeddings for “the cat sat” and “sat cat the”, the multiset of token embeddings might look similar. Without extra information, the model has no explicit “slot index” to tell it which token was first.

Positional encoding fixes this by injecting order information into the representation.

Core definition

Let p be a position index (typically p ∈ {0, 1, …, L−1} for a sequence length L). A positional encoding is a mapping

• •PE : p ↦ PE(p) ∈ ℝᵈ

where d is the model’s embedding dimension.

Each token also has an embedding (from a learned embedding table):

• •E(token) ∈ ℝᵈ

To create the model input at position p, we combine them, often by simple addition:

• •xₚ = E(tokenₚ) + PE(p)

Here xₚ, E(tokenₚ), and PE(p) are vectors in ℝᵈ. (We’ll write vectors in bold: x, v.)

What properties do we want?

Positional encoding should make position usable for attention and downstream layers. In practice, that means:

1. 1)Distinctness: different positions should map to different vectors.
2. 2)Smoothness / structure: nearby positions should relate in a learnable way (useful for local patterns).
3. 3)Generalization: ideally it should work for sequence lengths longer than those seen in training.
4. 4)Compatibility with attention: attention computes dot products like (qᵢ)ᵀ(kⱼ); we want positional info to influence these comparisons.

Absolute vs relative: the big split

There are two main ways to represent order:

• •Absolute positional information: “This token is at position p.” (PE(p) is tied to p.)
• •Relative positional information: “Token j is (j−i) steps away from token i.” (Encoding depends on offset.)

Both can work, but they lead to different inductive biases and different generalization behavior.

How it’s integrated (big picture)

There are two common integration mechanisms:

1. 1)Add/concatenate to embeddings (absolute-style):

• •xₚ = E(tokenₚ) + PE(p)

2. 2)Attention bias / modification (relative-style):

• •attention score(i, j) = (qᵢ)ᵀ(kⱼ) / √dₖ + b(i, j)
• •where b(i, j) depends on relative position (j−i) or sometimes absolute indices.

A useful mental model:

• •Absolute encoding changes “what each token looks like.”
• •Relative encoding changes “how tokens compare to each other.”

Motivation: give every slot an identity

Absolute positional encodings make position a feature of the token representation. If position p has a unique “signature” vector PE(p), then the model can learn rules like:

• •“Attend strongly to position 0 (start token).”
• •“Treat positions near the end differently.”
• •“If token is at position p=1, it’s likely a subject (in some domains).”

The simplest integration is addition:

• •xₚ = E(tokenₚ) + PE(p)

Addition is popular because it preserves dimension d and is parameter-free at the integration step.

Learned absolute positional embeddings

The most direct approach is to learn a table P ∈ ℝᴸmax×ᵈ, where Lmax is the maximum length during training.

• •PE(p) = P[p]

Pros:

• •Very flexible: the model can discover whatever positional geometry helps.

Cons:

• •Hard cap at Lmax unless you extend/resize.
• •Often weak extrapolation to longer sequences (positions beyond training are unseen).

Sinusoidal positional encoding (classic Transformer)

Sinusoidal PE is deterministic and defined by sines and cosines of different frequencies. For embedding dimension d (assume d is even), define for p ≥ 0 and i ∈ {0, 1, …, d/2 − 1}:

• •PE(p)[2i] = sin(p / 10000^(2i/d))
• •PE(p)[2i+1] = cos(p / 10000^(2i/d))

This produces a length-d vector with alternating sine/cosine pairs.

Why sinusoids?

The key intuition is multi-scale periodic structure:

• •Small i (low frequency denominator) varies quickly with p → captures local position changes.
• •Large i varies slowly → captures coarse position.

The encoding gives each position a unique pattern across many frequencies.

A useful property: relative shifts become linear-ish

A major reason sinusoids were chosen is that relative position relationships can be expressed via trig identities.

Let ωᵢ = 1 / 10000^(2i/d). Then:

• •sin((p+Δ)ωᵢ) = sin(pωᵢ)cos(Δωᵢ) + cos(pωᵢ)sin(Δωᵢ)
• •cos((p+Δ)ωᵢ) = cos(pωᵢ)cos(Δωᵢ) − sin(pωᵢ)sin(Δωᵢ)

So the pair [sin(pωᵢ), cos(pωᵢ)] can be transformed to [sin((p+Δ)ωᵢ), cos((p+Δ)ωᵢ)] by a 2×2 rotation matrix depending only on Δ.

That matters because attention compares vectors via dot products. If your representation contains sinusoidal components, the model can learn to detect or use relative offsets.

Table: absolute options compared

Practical details: scaling and dropout

In many implementations, token embeddings are scaled by √d before adding PE:

• •xₚ = √d · E(tokenₚ) + PE(p)

Why? Early in training, learned embeddings may have small variance; scaling can help balance magnitudes so positional info doesn’t dominate (or vanish).

Also, dropout is often applied to the sum:

• •xₚ = Dropout( √d · E(tokenₚ) + PE(p) )

This regularizes reliance on any single dimension or exact positional signature.

Limitation of pure absolute encoding

Absolute encodings label positions, but attention often cares about relative patterns:

• •“Look one token to the left.”
• •“Attend to the previous occurrence.”

Absolute encodings can still learn these behaviors, but it can be less direct, especially when you want translation-invariant rules (“the same rule anywhere in the sequence”). That motivates relative positional mechanisms.

Motivation: attention is about relationships

Self-attention’s core operation for a query position i and key position j is a similarity score:

• •score(i, j) = (qᵢ)ᵀ(kⱼ) / √dₖ

This has no explicit awareness of distance or direction (left vs right). Relative positional encodings inject information about (j−i) directly into the scoring.

A common pattern:

• •score(i, j) = (qᵢ)ᵀ(kⱼ) / √dₖ + b(j−i)

where b(Δ) is a learned scalar bias for offset Δ (possibly clipped to a range).

Relative position as an embedding

Instead of a scalar bias, we can use a vector embedding R[Δ] ∈ ℝᵈ and incorporate it into attention:

One family of formulations modifies keys/values:

• •score(i, j) = (qᵢ)ᵀ( kⱼ + r_{j−i} ) / √dₖ

Expand this:

• •(qᵢ)ᵀ( kⱼ + r_{j−i} )

= (qᵢ)ᵀkⱼ + (qᵢ)ᵀr_{j−i}

So the score becomes content similarity plus an offset-dependent term.

Why this is powerful

Relative encoding bakes in a helpful inductive bias:

• •If the same local pattern occurs at different absolute locations, the model can reuse it.
• •The model can naturally learn “nearby tokens matter more” or directional rules.

It also often improves generalization to longer sequences because offsets (like Δ = ±1, ±2, …) are length-agnostic.

Clipping and windowing offsets

In practice, you can’t store embeddings for every possible offset if sequences are long. Typical strategies:

• •Clip Δ into [−K, K]
• •Use a bucket function that maps large distances into coarse bins

So:

• •Δ′ = clip(j−i, −K, K)
• •use b(Δ′) or r_{Δ′}

This trades exact long-range distance for manageable parameters.

Rotary positional embeddings (RoPE) as a relative-ish approach

Another widely used modern method is to apply a position-dependent rotation to queries and keys so that dot products encode relative offsets.

High-level idea:

• •Split dimensions into 2D pairs.
• •For each position p, rotate each pair by an angle θ(p) that depends on p.
• •Use rotated queries/keys in attention.

For one 2D pair, if u = [u₀, u₁]ᵀ, define rotation:

• •R(θ) = [[cos θ, −sin θ],

[sin θ, cos θ]]

Then u′ = R(θ) u.

If queries and keys are rotated by their positions, the dot product between rotated vectors can become a function of relative angle differences (hence relative position). This tends to generalize well and is popular in LLMs.

Table: relative vs absolute (conceptual)

Integration mechanism summary

You can think of positional information entering the model in two places:

1. 1)Input space: modify xₚ = E(tokenₚ) + PE(p)
2. 2)Attention space: modify score(i, j)

Many strong architectures use both (or a more sophisticated variant) because input position and attention distance can each be useful.

Where it sits in the Transformer pipeline

A simplified Transformer input pipeline looks like:

1. 1)Tokenize text → tokens t₀…t_{L−1}
2. 2)Embed tokens → E(tₚ) ∈ ℝᵈ
3. 3)Add positional encoding → xₚ = E(tₚ) + PE(p)
4. 4)Feed xₚ into stacked self-attention + MLP blocks

Inside each attention layer (single head for simplicity):

• •qᵢ = W_Q xᵢ
• •kⱼ = W_K xⱼ
• •vⱼ = W_V xⱼ

Attention weights:

• •α(i, j) = softmaxⱼ( score(i, j) )

With basic attention:

• •score(i, j) = (qᵢ)ᵀ(kⱼ) / √dₖ

With relative bias:

• •score(i, j) = (qᵢ)ᵀ(kⱼ) / √dₖ + b(j−i)

Output:

• •yᵢ = ∑ⱼ α(i, j) vⱼ

What positional encoding enables (concrete behaviors)

Once the model has position information, it can learn:

• •Word order sensitivity: “dog bites man” vs “man bites dog”
• •Syntax patterns: verbs often follow subjects in English
• •Local features: n-gram-like patterns via attention to nearby positions
• •Segment structure: headers, lists, code blocks, etc.

Choosing a method: engineering trade-offs

Connection to the next node: Transformers

Positional encoding is one of the key ingredients that make Transformers work on sequences at all. Without it, self-attention is permutation-invariant: it can’t reliably distinguish different orderings. Once you add PE, the attention mechanism becomes sequence-aware.

In the Transformers node, you’ll see how multi-head attention and stacked layers exploit this positional information to build hierarchical features—from local dependencies to global structure.