Adapter Methods & Parameter-Efficient Fine-Tuning

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. LoRA: Low-Rank Adaptation
5. QLoRA: Quantized LoRA
6. AdaLoRA: Adaptive Rank
7. Prefix Tuning
8. Prompt Tuning
9. (IA)3: Few-Parameter Adaptation
10. Choosing the Right Method
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Derive LoRA from the low-rank hypothesis of weight updates.
2. Explain QLoRA's memory savings via NF4 quantization.
3. Compare prefix tuning, prompt tuning, and adapter methods.
4. Analyze when full fine-tuning outperforms PEFT.
5. Select the optimal PEFT method for a given scenario.

Notation

• $\Delta\mathbf{W} = \mathbf{BA}$ — LoRA decomposition ($\mathbf{B} \in \mathbb{R}^{d \times r}, \mathbf{A} \in \mathbb{R}^{r \times k}$)
• $r$ — rank (LoRA rank)
• $\alpha$ — LoRA scaling factor

Core Intuition

Full fine-tuning updates all parameters ($\sim$billions) but typically changes each weight by only a small amount. This suggests the weight UPDATE $\Delta\mathbf{W}$ has low rank — it lies in a low-dimensional subspace. PEFT methods exploit this by parameterizing the update efficiently: LoRA uses rank-$r$ matrices, prefix tuning adds learnable tokens, prompt tuning modifies the input embedding.

LoRA: Low-Rank Adaptation

Forward pass:

where $\mathbf{W}_0$ is frozen, $\mathbf{A}$ initialized from $\mathcal{N}(0, \sigma^2)$, $\mathbf{B}$ initialized to zero.

Parameters: $r(d + k)$ per adapted layer vs $dk$ for full fine-tuning. For $r=16, d=4096, k=4096$: 131K vs 16.8M (128x reduction).

Applied to: Typically $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$ in attention. Optionally FFN layers.

Merging: After training, merge: $\mathbf{W} = \mathbf{W}_0 + \frac{\alpha}{r}\mathbf{BA}$. No inference overhead.

QLoRA: Quantized LoRA

Innovation: Combine 4-bit quantization of base model with LoRA adapters:

1. Quantize $\mathbf{W}_0$ to NF4 (4-bit NormalFloat).
2. Add FP16 LoRA adapters on top.
3. Backpropagate through the quantized weights to update LoRA.

Memory: 70B model: $\sim$35 GB (NF4) + tiny LoRA adapters. Fits on single 48GB GPU.

Double quantization: Even the quantization constants (scales) are quantized to FP8, saving additional memory.

AdaLoRA: Adaptive Rank

Problem: Not all layers need the same rank. Important layers benefit from higher rank; less important ones need minimal adaptation.

Solution: Parameterize adapters with SVD and prune small singular values:

Rank is allocated adaptively based on importance (measured by sensitivity of the loss to that component).

Prefix Tuning

Add learnable "virtual tokens" to keys and values:

where $\mathbf{P}_K, \mathbf{P}_V \in \mathbb{R}^{l \times d}$ are learned prefix embeddings ($l$ = prefix length).

Parameters: $2ld$ per layer. For $l=20, d=4096, L=32$: 5.2M total.

Advantage: Modular — different prefixes for different tasks, swappable at inference.

Prompt Tuning

Simplest PEFT: Only add learnable embeddings at the input layer:

Parameters: $ld$ total ($\sim$80K for $l=20, d=4096$).

Performance: Competitive with full fine-tuning only for very large models (above 10B). For smaller models, insufficient expressiveness.

(IA)3: Few-Parameter Adaptation

Learned rescaling vectors for keys, values, and FFN:

Parameters: Only $3d$ per layer ($\sim$400K total for a 7B model). Extremely lightweight.

Performance: Surprisingly effective for classification tasks. Less effective for generation.

Choosing the Right Method

• Full fine-tuning: When you have enough compute and data, and need maximum quality.
• LoRA ($r=16$–$64$): Default choice for instruction tuning and domain adaptation.
• QLoRA: When GPU memory is the constraint (single GPU for large models).
• Prefix tuning: When you need multiple task-specific modules (swappable at runtime).
• Prompt tuning: For very large models with minimal parameter budget.
• (IA)3: For classification/NLU tasks with extreme parameter efficiency.

Common Pitfalls

Pitfall 1. Using too low rank for complex tasks. $r=4$ works for simple classification; instruction tuning often needs $r=32$–$64$.

Pitfall 2. Only applying LoRA to attention layers. For best results, also adapt FFN layers (up and down projections).

Pitfall 3. Not tuning the LoRA $\alpha/r$ ratio. This controls the effective learning rate for the adapter. Default $\alpha=r$ works but isn't always optimal.

Summary

• LoRA: Low-rank weight update; 128x fewer parameters; no inference overhead after merging.
• QLoRA: LoRA on 4-bit base model; enables 70B fine-tuning on single GPU.
• AdaLoRA: Adaptive rank allocation based on layer importance.
• Prefix/Prompt tuning: Learnable virtual tokens; modular and task-swappable.
• (IA)3: Minimal parameters (rescaling only); good for classification.
• Choice depends on: quality needs, memory budget, modularity requirements.

Exercises

Exercise 1. For LoRA with $r=32$ on all attention layers of a 7B model ($d=4096, L=32$): compute total trainable parameters and percentage of full model.

Exercise 2. Derive the gradient of the loss with respect to LoRA matrix $\mathbf{A}$ (showing it flows through frozen $\mathbf{W}_0$).

Exercise 3. Compare memory requirements for QLoRA ($r=64$) vs full fine-tuning of a 13B model with AdamW.

Exercise 4. Design an experiment to determine the optimal rank $r$ for fine-tuning on a specific task.

Exercise 5. Explain why LoRA with $\mathbf{B}$ initialized to zero ensures the model starts at the pre-trained weights.