ML

Rotary Position Embeddings (RoPE)

Volume III, Chapter 11 — Part II. Complete derivation of RoPE: rotation matrices, relative position in attention scores, frequency spectrum, length extrapolation, and NTK-aware scaling.

Intermediate

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Prerequisites
  3. Notation
  4. Core Intuition
  5. Rotation Matrices and 2D Blocks
  6. RoPE Definition
  7. Relative Position Property
  8. Frequency Spectrum and Distance Decay
  9. Implementation Structure
  10. Length Extrapolation and NTK Scaling
  11. Comparison with Other Position Methods
  12. Worked Examples
  13. Connection to the Broader Curriculum
  14. Common Pitfalls and Misconceptions
  15. Research Perspective
  16. Summary of Takeaways
  17. Exercises

Learning Objectives

After reading this chapter, you should be able to:

  1. Define RoPE as block-diagonal rotation matrices applied to Q and K.
  2. Prove q~tTk~t′=qtTRt′−tkt′\tilde{\mathbf{q}}_t^T \tilde{\mathbf{k}}_{t'} = \mathbf{q}_t^T \mathbf{R}_{t'-t} \mathbf{k}_{t'}.
  3. Explain the geometric frequency spectrum and long-range decay.
  4. Describe NTK-aware scaling for context extension.
  5. Explain RoPE compatibility with KV Cache.

Prerequisites

Notation

  • m,nm, n — Absolute positions in sequence
  • dd — Head dimension (assumed even)
  • θi=10000−2i/d\theta_i = 10000^{-2i/d} — Rotary frequency at dimension pair ii
  • Rm\mathbf{R}_m — Rotation matrix applied at position mm
  • qm,kn\mathbf{q}_m, \mathbf{k}_n — Query and key vectors with RoPE applied

Core Intuition

Positional Encoding showed that encoding position in Q/K dot products enables relative position awareness. RoPE (Su et al., 2021) applies this directly: rotate Q and K by position-dependent angles before computing attention.

The attention score depends only on relative position t′−tt' - t, not absolute indices. RoPE requires zero extra parameters, works with KV Cache, and extrapolates to longer sequences with appropriate scaling — making it the standard in LLaMA, Mistral, and GPT-NeoX.

Series context. Volume III, Chapter 11, Part II. Extends Positional Encoding.

Rotary Position Embeddings (RoPE)

Q (pos 0)K (pos 4)θ = m·ωRelative distance4Q·K dot product0.000Decay ≈ cos(θ)0.000distance →
Distance
4
Query vectorKey vector
Explore: RoPE rotates Q and K by position-dependent angles. The dot product depends only on relative distance — attention naturally decays for distant tokens.

Rotation Matrices and 2D Blocks

Definition 1 (2D Rotation).

R(θ)=[cos⁡θ−sin⁡θsin⁡θcos⁡θ].(1)\mathbf{R}(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}. \tag{1}

Proposition 1. R(θ1)TR(θ2)=R(θ2−θ1)\mathbf{R}(\theta_1)^T \mathbf{R}(\theta_2) = \mathbf{R}(\theta_2 - \theta_1).

Proof. Rotation composition adds angles; transpose inverts rotation. ■\blacksquare

RoPE Definition

Definition 2 (RoPE Frequencies).

ωi=θbase−2i/dk,i=0,1,…,dk/2−1.(2)\omega_i = \theta_{\text{base}}^{-2i/d_k}, \quad i = 0, 1, \ldots, d_k/2 - 1. \tag{2}

Typical θbase=10000\theta_{\text{base}} = 10000.

Definition 3 (RoPE Transform). For position tt, apply block-diagonal rotation:

Rt=diag(R(ω0t),R(ω1t),…,R(ωdk/2−1t)).(3)\mathbf{R}_t = \text{diag}(\mathbf{R}(\omega_0 t), \mathbf{R}(\omega_1 t), \ldots, \mathbf{R}(\omega_{d_k/2-1} t)). \tag{3} q~t=Rtqt,k~t′=Rt′kt′.(4)\tilde{\mathbf{q}}_t = \mathbf{R}_t \mathbf{q}_t, \quad \tilde{\mathbf{k}}_{t'} = \mathbf{R}_{t'} \mathbf{k}_{t'}. \tag{4}

Attention uses rotated Q, K; V is unchanged.

Relative Position Property

Theorem 1 (RoPE Relative Attention).

q~tTk~t′=qtTRt′−tkt′.(5)\tilde{\mathbf{q}}_t^T \tilde{\mathbf{k}}_{t'} = \mathbf{q}_t^T \mathbf{R}_{t'-t} \mathbf{k}_{t'}. \tag{5}

Proof.

q~tTk~t′=qtTRtTRt′kt′=qtTRt′−tkt′.(6)\tilde{\mathbf{q}}_t^T \tilde{\mathbf{k}}_{t'} = \mathbf{q}_t^T \mathbf{R}_t^T \mathbf{R}_{t'} \mathbf{k}_{t'} = \mathbf{q}_t^T \mathbf{R}_{t'-t} \mathbf{k}_{t'}. \tag{6}

by Proposition 1. ■\blacksquare

Important equation. Attention depends on (t′−t)(t' - t) only — native relative position encoding.

Frequency Spectrum and Distance Decay

Proposition 2. High-frequency components (ωi\omega_i large) encode fine local position; low-frequency components encode coarse global position.

Proposition 3. For large ∣t−t′∣|t - t'|, high-frequency terms oscillate rapidly, reducing correlation — implicit distance decay.

Analogous to sinusoidal PE in Positional Encoding.

Implementation Structure

RoPE acts on consecutive dimension pairs (2i,2i+1)(2i, 2i+1):

[q2i′q2i+1′]=[cos⁡(ωit)−sin⁡(ωit)sin⁡(ωit)cos⁡(ωit)][q2iq2i+1].(7)\begin{bmatrix} q_{2i}' \\ q_{2i+1}' \end{bmatrix} = \begin{bmatrix} \cos(\omega_i t) & -\sin(\omega_i t) \\ \sin(\omega_i t) & \cos(\omega_i t) \end{bmatrix} \begin{bmatrix} q_{2i} \\ q_{2i+1} \end{bmatrix}. \tag{7}

For cached K: store Rt′kt′\mathbf{R}_{t'} \mathbf{k}_{t'} or apply rotation at cache-write time with position index.

Length Extrapolation and NTK Scaling

Problem. RoPE trained on length LL may degrade at L′>LL' > L.

Definition 4 (NTK-Aware Scaling). Rescale base frequency at inference:

ωi′=ωi/s2i/dk,s=L′/L.(8)\omega_i' = \omega_i / s^{2i/d_k}, \quad s = L'/L. \tag{8}

Interpretation. Interpolates between frequencies to maintain attention coherence at extended context.

Related: Position interpolation (PI), YaRN — active research for million-token contexts.

Comparison with Other Position Methods

See Positional Encoding comparison table. RoPE advantages:

  • Zero parameters
  • Native relative encoding
  • KV-cache compatible
  • Better extrapolation than learned absolute PE

Worked Examples

Example 1: Relative Invariance

Attention between positions (5,8)(5, 8) equals (105,108)(105, 108) — both distance 3.

Example 2: Rotation Angle

At t=1000t = 1000, ω0t\omega_0 t may exceed 2π2\pi — periodicity wraps position information.

Connection to the Broader Curriculum

Common Pitfalls and Misconceptions

Pitfall 1: Applying RoPE to V (incorrect — only Q and K).

Pitfall 2: Wrong position index during cached decode.

Pitfall 3: Assuming unlimited extrapolation without scaling.

Research Perspective

RoPE (Su et al., 2021). NTK scaling (bloc97, 2023). YaRN, LongRoPE. Alternatives: ALiBi, CoPE.

Summary of Takeaways

  • RoPE — q~t=Rtqt\tilde{\mathbf{q}}_t = \mathbf{R}_t \mathbf{q}_t
  • Relative — qtTRt′−tkt′\mathbf{q}_t^T \mathbf{R}_{t'-t} \mathbf{k}_{t'}
  • Frequencies — ωi=θbase−2i/dk\omega_i = \theta_{\text{base}}^{-2i/d_k}
  • Extrapolation — NTK scaling

Next: Scaling Laws

Exercises

Exercise 1. Prove Proposition 1.

Exercise 2. Prove Theorem 1.

Exercise 3. Derive (7) for single pair.

Exercise 4. Why is V not rotated?

Exercise 5. KV cache: when to apply Rt\mathbf{R}_t?

Exercise 6. Compare RoPE to sinusoidal PE addition.

Exercise 7. Effect of θbase\theta_{\text{base}} on frequency spectrum.

Exercise 8. Derive NTK scaling motivation heuristically.

Next

Scaling Laws for Language Models