LoRA: Low-Rank Adaptation

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. The Full Fine-Tuning Problem
6. Low-Rank Hypothesis
7. LoRA Formulation
8. Parameter Count and Efficiency
9. Where to Apply LoRA
10. Training and Merging
11. Connection to SVD
12. Worked Examples
13. Connection to the Broader Curriculum
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. State the LoRA weight update $\mathbf{W}' = \mathbf{W} + \mathbf{B}\mathbf{A}$ with rank-$r$ decomposition.
2. Compute LoRA parameter count vs. full fine-tuning.
3. Explain the low-rank hypothesis for adaptation tasks.
4. Identify which layers (Q/K/V/O projections) LoRA typically targets.
5. Describe weight merging at inference: $\mathbf{W}' = \mathbf{W} + \mathbf{B}\mathbf{A}$.

Prerequisites

• SVD — low-rank approximation
• Backpropagation — gradient flow
• Self-Attention — projection matrices

Notation

• $\mathbf{W}_0 \in \mathbb{R}^{d \times k}$ — Frozen pretrained weight
• $\mathbf{B} \in \mathbb{R}^{d \times r}, \mathbf{A} \in \mathbb{R}^{r \times k}$ — Low-rank adapter factors
• $r$ — LoRA rank ($r \ll \min(d,k)$)
• $\alpha$ — LoRA scaling hyperparameter
• $\Delta \mathbf{W} = \mathbf{B}\mathbf{A}$ — Low-rank weight update

Core Intuition

Fine-tuning a 70B-parameter LLM requires updating all weights — prohibitive in memory and storage. LoRA (Hu et al., 2021) freezes pretrained weights $\mathbf{W}$ and learns a low-rank update $\Delta \mathbf{W} = \mathbf{B}\mathbf{A}$ where $\mathbf{B} \in \mathbb{R}^{d \times r}$, $\mathbf{A} \in \mathbb{R}^{r \times k}$, $r \ll \min(d,k)$.

The hypothesis: task-specific adaptation lives in a low-dimensional subspace of weight space — supported by intrinsic dimension studies and SVD theory.

Series context. Volume III, Chapter 14 (Fine-Tuning).

The Full Fine-Tuning Problem

Definition 1 (Full Fine-Tuning). Update all parameters $\boldsymbol{\theta}$ of pretrained model:

Proposition 1. For $|\boldsymbol{\theta}| = N$, fine-tuning requires storing $N$ optimizer states + $N$ gradients — typically $3N$ additional memory.

For $N = 70 \times 10^9$, FP16: $\sim 420$ GB — infeasible on consumer hardware.

Low-Rank Hypothesis

Definition 2 (Intrinsic Rank). The effective dimension of task-specific weight updates is much smaller than ambient dimension.

Proposition 2 (Eckart–Young). Best rank-$r$ approximation to $\Delta \mathbf{W}$ in Frobenius norm comes from top-$r$ SVD components.

LoRA parameterizes $\Delta \mathbf{W}$ directly as low-rank product rather than computing SVD post-hoc.

LoRA Formulation

Definition 3 (LoRA Layer). For linear layer $\mathbf{W}_0 \in \mathbb{R}^{d \times k}$:

where $\mathbf{B} \in \mathbb{R}^{d \times r}$, $\mathbf{A} \in \mathbb{R}^{r \times k}$, rank $r \ll \min(d, k)$.

Initialization: $\mathbf{A}$ random Gaussian, $\mathbf{B} = \mathbf{0}$ — so $\Delta \mathbf{W} = \mathbf{0}$ at start (pretrained behavior preserved).

Scaling: Often $\Delta \mathbf{W} \mathbf{x} \leftarrow (\alpha/r) \mathbf{B}\mathbf{A}\mathbf{x}$ with hyperparameter $\alpha$.

Parameter Count and Efficiency

Proposition 3. LoRA parameters for one layer: $r(d + k)$ vs. full: $dk$.

Example. $d = k = 4096$, $r = 16$: LoRA $= 16 \times 8192 = 131$K vs. full $= 16.8$M — 128× reduction per layer.

For entire Transformer: apply to Q/K/V/O projections across $L$ layers.

Where to Apply LoRA

Standard targets: $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$ in Self-Attention.

Optional: MLP layers ($\mathbf{W}_{\text{up}}, \mathbf{W}_{\text{down}}$), embedding layer.

Proposition 4. Applying LoRA to all attention projections captures most adaptation benefit at modest parameter cost.

Training and Merging

Training: Only $\mathbf{A}, \mathbf{B}$ receive gradients; $\mathbf{W}_0$ frozen.

Inference merge:

Merged model has zero inference overhead — same architecture as base model.

Multi-adapter: Keep separate $(\mathbf{B}_i, \mathbf{A}_i)$ per task; swap at runtime without merging.

Connection to SVD

Proposition 5. If optimal $\Delta \mathbf{W}^*$ has rank $r^*$, LoRA with $r \geq r^*$ can represent $\Delta \mathbf{W}^*$ exactly.

Proposition 6. LoRA is a reparameterization of rank-$r$ matrices via product manifold — not unique factorization (unlike SVD).

See SVD for optimal low-rank approximation theory.

Worked Examples

Example 1: Parameter Savings

Llama 7B: 32 layers, 4 projections, $d = 4096$, $r = 8$. LoRA params $\approx 32 \times 4 \times 8 \times 8192 \approx 8.4$M vs. 7B full.

Example 2: Merge

After training, $\mathbf{W}' = \mathbf{W}_0 + \mathbf{B}\mathbf{A}$ — single matrix multiply at inference.

Connection to the Broader Curriculum

• SVD — theoretical basis
• DPO — LoRA for alignment
• Self-Attention — target layers
• Quantization — composable with QLoRA

Common Pitfalls and Misconceptions

Pitfall 1: $r$ too small → underfitting; too large → no savings.

Pitfall 2: Forgetting scaling factor $\alpha/r$.

Pitfall 3: LoRA on all layers may not beat selective application.

Pitfall 4: Merged weights lose adapter modularity.

Research Perspective

LoRA (Hu et al., 2021). QLoRA (Dettmers et al., 2023). DoRA, AdaLoRA, VeRA. Standard for DPO and instruction tuning.

Summary of Takeaways

• LoRA — $\mathbf{W}' = \mathbf{W}_0 + \mathbf{B}\mathbf{A}$
• Params — $r(d + k)$ per layer
• Merge — Zero inference overhead
• Basis — Low-rank / SVD

Next: KV Cache

Exercises

Exercise 1. Compute LoRA params for given $d, k, r, L$.

Exercise 2. Prove $\text{rank}(\mathbf{B}\mathbf{A}) \leq r$.

Exercise 3. Why initialize $\mathbf{B} = \mathbf{0}$?

Exercise 4. Compare LoRA to top-$r$ SVD of full $\Delta \mathbf{W}$.

Exercise 5. Gradient flow: which params update in LoRA?

Exercise 6. LoRA + Quantization (QLoRA concept).

Exercise 7. Apply LoRA to Self-Attention Q projection only.

Exercise 8. Storage for 10 task-specific LoRA adapters vs. 10 full models.