Compression via Distillation & Low-Rank Factorization

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Low-Rank Factorization
5. SVD-Based Compression
6. LoRA as Inference Compression
7. Weight Tying & Parameter Sharing
8. Vocabulary Reduction
9. Progressive Compression
10. Combined Compression Strategies
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Apply SVD to decompose weight matrices with controlled rank.
2. Explain how LoRA enables post-hoc compression of fine-tuned models.
3. Design vocabulary reduction strategies for domain-specific deployment.
4. Construct progressive compression pipelines (pruning + quantization + distillation).
5. Estimate quality-compression tradeoffs for a given model and budget.

Notation

• $\mathbf{W} \in \mathbb{R}^{m \times n}$ — weight matrix
• $r$ — target rank for decomposition
• $\sigma_i$ — $i$-th singular value

Core Intuition

A trained weight matrix $\mathbf{W}$ is often approximately low-rank — most of its "information" lives in a small number of directions. By decomposing $\mathbf{W} \approx \mathbf{U}\mathbf{V}^T$ with rank $r \ll \min(m,n)$, we reduce parameters from $mn$ to $r(m+n)$ — potentially 4-8x compression with minimal quality loss. Combined with pruning and quantization, we achieve 10-50x total compression.

Low-Rank Factorization

Replace $\mathbf{W} \in \mathbb{R}^{m \times n}$ with two smaller matrices:

Parameter reduction:

• Original: $mn$ parameters.
• Factorized: $r(m+n)$ parameters.
• Compression ratio: $\frac{mn}{r(m+n)}$.
• For $m=n=4096, r=256$: compression = $\frac{4096^2}{256 \times 8192} = 8\times$.

Compute reduction: $\mathbf{W}\mathbf{x}$ costs $mn$ FLOPs. $\mathbf{A}(\mathbf{B}\mathbf{x})$ costs $rn + rm = r(m+n)$ — same compression ratio.

SVD-Based Compression

Truncated SVD: $\mathbf{W} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T \approx \mathbf{U}_r\boldsymbol{\Sigma}_r\mathbf{V}_r^T$.

Optimal rank-r approximation (Eckart-Young theorem):

Explained variance ratio:

Practical: Choose $r$ such that EVR$(r) > 0.99$ (retain 99% of information).

Per-layer rank selection: Different layers may have different effective ranks. Sensitivity analysis: measure loss increase per layer when compressed to rank $r$.

LoRA as Inference Compression

During fine-tuning: $\mathbf{W}' = \mathbf{W}_0 + \Delta\mathbf{W} = \mathbf{W}_0 + \mathbf{B}\mathbf{A}$.

For deployment: Merge: $\mathbf{W}_{\text{merged}} = \mathbf{W}_0 + \mathbf{B}\mathbf{A}$.

• No inference overhead (just a standard dense layer).
• But: full parameter count restored.

Alternative — keep factored:

• Store $\mathbf{W}_0$ in low precision (INT4) + $\mathbf{B}\mathbf{A}$ in FP16.
• During inference: $\mathbf{y} = \mathbf{W}_0^{\text{INT4}}\mathbf{x} + \mathbf{B}(\mathbf{A}\mathbf{x})$.
• Quality of FP16 with memory closer to INT4.

QLoRA insight: Quantize base weights aggressively; keep LoRA in full precision. The LoRA compensates for quantization error.

Weight Tying & Parameter Sharing

Embedding tying: Share input embedding and output projection:

For vocabulary $V=128K$, dimension $d=4096$: saves $V \times d = 512M$ parameters.

Cross-layer sharing (ALBERT): All transformer layers share parameters:

• 12-layer BERT → 1 set of parameters applied 12 times.
• 12x parameter reduction.
• Quality: 5-10% degradation but with 12x fewer parameters.

Partial sharing: Share attention parameters across layers, unique FFN per layer. Better quality/compression tradeoff.

Vocabulary Reduction

Problem: Vocabulary embedding is often 10-20% of total parameters. For multilingual models, many tokens are irrelevant for specific deployments.

Domain-specific reduction:

1. Analyze target domain token frequency.
2. Remove tokens with zero frequency in domain.
3. Retrain embedding for remaining tokens.

Example: 128K general vocabulary → 32K domain-specific:

• Embedding reduction: 4x (512M → 128M parameters for 4096-dim).
• Quality: maintained for domain tasks; degraded for out-of-domain.

Progressive Compression

Apply compression techniques sequentially, each building on previous:

Recommended order:

1. Structured pruning (remove channels/heads) → smaller architecture.
2. Low-rank factorization (SVD on remaining layers) → fewer parameters.
3. Quantization (INT8/INT4 on factored model) → smaller memory.
4. Distillation (fine-tune compressed model with teacher guidance) → recover quality.

Each step: Apply → fine-tune → validate. Then proceed to next.

Typical results (70B → deployment):

• Pruning (50% heads): 70B → 40B effective.
• Low-rank (r=512): 40B → 25B parameters.
• Quantization (INT4): 25B × 4B → 12.5GB memory.
• Distillation: recover 1-2% lost accuracy.
• Total: 6x memory reduction with less than 3% quality loss.

Combined Compression Strategies

Common Pitfalls

Pitfall 1. Applying SVD without fine-tuning afterward. Raw SVD compression causes noticeable quality loss. Always fine-tune the factored model for a few hundred steps.

Pitfall 2. Same rank for all layers. Attention layers typically need higher rank (more complex transformations) than FFN layers (more redundant). Use per-layer rank selection.

Pitfall 3. Vocabulary reduction without verifying tokenization coverage. If important domain terms become out-of-vocabulary after reduction, the model can't process them. Always check coverage on domain test set.

Summary

• Low-rank factorization: $\mathbf{W} \approx \mathbf{AB}$; 4-8x compression for rank-deficient layers.
• SVD: Optimal rank-$r$ approximation; choose $r$ by explained variance.
• LoRA for deployment: Keep base in INT4, adapter in FP16.
• Weight tying: Share embeddings across input/output; save 10-20% parameters.
• Vocabulary reduction: Remove unused tokens for domain deployment.
• Progressive compression: Prune → factorize → quantize → distill (10-30x total).

Exercises

Exercise 1. For a 4096×4096 weight matrix with singular values decaying as $\sigma_i = 100/i$: find the rank $r$ needed for 99% explained variance.

Exercise 2. Compare the memory and compute of: (a) dense 4096×4096 in FP16, (b) factored rank-512 in FP16, (c) dense 4096×4096 in INT4.

Exercise 3. Design a progressive compression pipeline for LLaMA-7B targeting deployment on a smartphone (4GB RAM). Specify each step and expected quality.

Exercise 4. For ALBERT-style cross-layer sharing: prove that the effective capacity is NOT reduced by the sharing factor (layers can still learn different functions via different inputs).

Exercise 5. Compare vocabulary reduction strategies for deploying a general LLM to: (a) medical domain, (b) legal domain, (c) code-only. How much parameter savings in each case?