Positional Encoding & RoPE

Learning Objectives
Notation
Core Intuition
Why Position Matters
Sinusoidal Positional Encoding
Learned Positional Embeddings
Rotary Position Embedding (RoPE)
ALiBi (Attention with Linear Biases)
Length Generalization
Common Pitfalls
Summary
Exercises

Learning Objectives

Prove that attention without positional encoding is permutation-invariant.
Derive sinusoidal positional encoding and explain the geometric intuition.
Derive RoPE from the relative position kernel requirement.
Compare absolute vs relative position methods.
Explain how RoPE enables context length extension.

Notation

$\text{pos}$ — absolute position index
$\omega_k = 1/10000^{2k/d}$ — frequency for dimension $k$
$\mathbf{R}_\theta$ — rotation matrix for RoPE
$m$ — slope parameter in ALiBi

Core Intuition

Self-attention treats its input as a set — it is permutation-equivariant by construction. This means without positional information, "The cat sat on the mat" and "mat the on sat cat The" produce identical attention patterns. Positional encoding breaks this symmetry, telling the model where each token is.

Rotary Position Embeddings (RoPE)

Distance

Query vectorKey vector

Explore: RoPE rotates Q and K by position-dependent angles. The dot product depends only on relative distance — attention naturally decays for distant tokens.

Why Position Matters

Theorem. Self-attention is permutation-equivariant. For any permutation matrix $\mathbf{P}$ :

\text{Attention}(\mathbf{PX}) = \mathbf{P} \cdot \text{Attention}(\mathbf{X}). \tag{1}

Proof. Let $\mathbf{Q}' = \mathbf{PXW}^Q = \mathbf{PQ}$ , similarly $\mathbf{K}' = \mathbf{PK}$ , $\mathbf{V}' = \mathbf{PV}$ .

\text{softmax}(\mathbf{Q}'\mathbf{K}'^T/\sqrt{d_k})\mathbf{V}' = \text{softmax}(\mathbf{PQK}^T\mathbf{P}^T/\sqrt{d_k})\mathbf{PV} = \mathbf{P}\text{softmax}(\mathbf{QK}^T/\sqrt{d_k})\mathbf{V}. \; \blacksquare

Thus, without positional encoding, word order has no effect on the computation.

Sinusoidal Positional Encoding

Original Transformer (2017): Add fixed sinusoidal vectors to token embeddings:

\text{PE}(\text{pos}, 2k) = \sin(\text{pos} \cdot \omega_k), \quad \text{PE}(\text{pos}, 2k+1) = \cos(\text{pos} \cdot \omega_k), \tag{2}

where $\omega_k = 1/10000^{2k/d}$ .

Properties:

Each dimension oscillates at a different frequency (from $2\pi$ to $2\pi \cdot 10000$ ).
Relative positions can be expressed as linear transformations: $\text{PE}(\text{pos}+k) = \mathbf{M}_k \cdot \text{PE}(\text{pos})$ for a matrix $\mathbf{M}_k$ depending only on $k$ .
Bounded norm regardless of position.

Geometric interpretation: Position $p$ is encoded as a point on a $d/2$ -dimensional torus, with each pair of dimensions tracing a circle at a different angular velocity.

Learned Positional Embeddings

Replace fixed sinusoids with a learnable lookup table:

\mathbf{x}_{\text{pos}} = \mathbf{e}_{\text{token}} + \mathbf{p}_{\text{pos}}, \quad \mathbf{p}_{\text{pos}} \in \mathbb{R}^d \text{ (learned)}. \tag{3}

Advantage: Can learn task-specific positional patterns. Disadvantage: Cannot generalize to positions unseen during training.

Used in: GPT-2, BERT.

Rotary Position Embedding (RoPE)

Key insight: Encode position into the dot-product directly, making the attention score depend only on relative position.

Requirement: Find a position encoding such that:

\langle f(\mathbf{q}, m), f(\mathbf{k}, n) \rangle = g(\mathbf{q}, \mathbf{k}, m - n), \tag{4}

where $m, n$ are absolute positions but the result depends only on the difference.

Solution (2D case): Apply a rotation of angle $\theta = m \cdot \omega$ to the query, and $\theta = n \cdot \omega$ to the key:

f(\mathbf{x}, p) = \begin{pmatrix}\cos(p\omega) & -\sin(p\omega) \\ \sin(p\omega) & \cos(p\omega)\end{pmatrix}\begin{pmatrix}x_1 \\ x_2\end{pmatrix} = \mathbf{R}_{p\omega}\mathbf{x}. \tag{5}

Then:

\langle f(\mathbf{q}, m), f(\mathbf{k}, n) \rangle = \mathbf{q}^T\mathbf{R}_{m\omega}^T\mathbf{R}_{n\omega}\mathbf{k} = \mathbf{q}^T\mathbf{R}_{(n-m)\omega}\mathbf{k}. \tag{6}

General $d$ dimensions: Apply independent 2D rotations to each consecutive pair of dimensions, with frequencies $\omega_k = \theta^{-2k/d}$ (where $\theta = 10000$ ):

\mathbf{R}_{\Theta, p} = \text{diag}(\mathbf{R}_{p\omega_0}, \mathbf{R}_{p\omega_1}, \ldots, \mathbf{R}_{p\omega_{d/2-1}}). \tag{7}

Properties:

Relative position encoding via absolute rotations.
No additional parameters.
Decays naturally with distance (due to oscillation).
Compatible with KV-cache (position applied at projection time).

Used in: LLaMA, Mistral, GPT-NeoX, most modern LLMs.

ALiBi (Attention with Linear Biases)

Approach: No positional embedding at all. Instead, add a linear bias to attention scores:

\text{score}(i, j) = \frac{\mathbf{q}_i^T\mathbf{k}_j}{\sqrt{d_k}} - m \cdot |i - j|, \tag{8}

where $m$ is a head-specific slope ( $m_h = 2^{-8h/H}$ for geometric spacing).

Effect: Penalizes distant tokens linearly. Closer tokens always have higher baseline scores.

Advantage: Excellent length generalization — trains on short sequences, works on longer ones.

Used in: BLOOM, MPT.

Length Generalization

Problem: Models trained on length $T_{\text{train}}$ degrade at length $T_{\text{test}} > T_{\text{train}}$ .

Methods for extending context:

RoPE frequency scaling (NTK-aware): Scale base frequency $\theta$ to accommodate longer sequences:

\theta' = \theta \cdot \left(\frac{T_{\text{test}}}{T_{\text{train}}}\right)^{d/(d-2)}. \tag{9}

YaRN: Combines NTK scaling with attention temperature scaling.
Position interpolation: Map positions $[0, T_{\text{test}}]$ to $[0, T_{\text{train}}]$ : use position $p \cdot T_{\text{train}}/T_{\text{test}}$ .

Common Pitfalls

Pitfall 1. Adding positional encoding to both Q and K and V. Standard practice adds position to the input (affecting all), but RoPE only rotates Q and K (not V). Applying to V would make the value content position-dependent, which is undesirable.

Pitfall 2. Assuming learned positional embeddings generalize. They strictly cannot: position 4097 has no embedding if training used max length 4096.

Pitfall 3. Extending RoPE context without fine-tuning. Raw extrapolation fails; some fine-tuning on longer data is needed after frequency adjustment.

Summary

Without positional encoding, attention is permutation-invariant.
Sinusoidal: Fixed frequencies, limited to training length.
Learned: Flexible but non-generalizable.
RoPE: Rotary encoding that injects relative position into dot products; dominant approach.
ALiBi: Linear bias penalty; excellent generalization.
Context extension: NTK scaling, YaRN, position interpolation.

Exercises

Exercise 1. Verify that $\mathbf{R}_\alpha^T\mathbf{R}_\beta = \mathbf{R}_{\beta-\alpha}$ for 2D rotation matrices.

Exercise 2. Compute the dot product $\langle \text{RoPE}(\mathbf{q}, 5), \text{RoPE}(\mathbf{k}, 3)\rangle$ for $\mathbf{q} = [1, 0], \mathbf{k} = [0, 1], \omega = \pi/4$ .

Exercise 3. Prove that sinusoidal positional encoding satisfies $\text{PE}(p+k) = M_k \cdot \text{PE}(p)$ for an explicit matrix $M_k$ .

Exercise 4. For ALiBi with $H=8$ heads, compute all slope values $m_h$ .

Exercise 5. Explain why position interpolation (scaling $p$ to $p \cdot T_{\text{train}}/T_{\text{test}}$ ) degrades less than extrapolation for RoPE.