Transfer Learning & Domain Adaptation

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Feature Extraction vs Fine-tuning
5. Domain Shift Theory
6. Unsupervised Domain Adaptation
7. Few-Shot Transfer Learning
8. The Foundation Model Paradigm
9. When Transfer Helps vs Hurts
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Decide between feature extraction and full fine-tuning for a given scenario.
2. Formalize domain shift and bound the target domain error.
3. Describe domain-adversarial networks for unsupervised adaptation.
4. Apply in-context learning as zero-shot transfer.
5. Identify when pre-training hurts (negative transfer).

Notation

• $\mathcal{D}_S, \mathcal{D}_T$ — source and target domains
• $d_{\mathcal{H}}$ — $\mathcal{H}$-divergence between domains
• $\epsilon_S, \epsilon_T$ — source and target error

Core Intuition

A model trained on ImageNet learns general visual features (edges, textures, shapes) that transfer to medical imaging, satellite imagery, or manufacturing inspection — even though these domains look very different. Transfer learning exploits this: start with pre-trained representations and adapt to the target domain with minimal data. The amount of adaptation (none, partial, full) depends on the domain gap and available target data.

Feature Extraction vs Fine-tuning

Feature extraction: Freeze pre-trained backbone; train only the final classifier:

• Pro: Works with very few examples (10-100).
• Pro: No risk of overwriting good features.
• Con: Limited adaptation to target domain.

Full fine-tuning: Update all parameters on target data:

• Pro: Maximum adaptation; best performance with sufficient data.
• Con: Needs more data (1000+); risk of overfitting/forgetting.

Partial fine-tuning: Freeze early layers (general features); fine-tune later layers (task-specific):

• Compromise between the two extremes.
• Common: freeze first 75% of layers.

Decision rule (empirical):

• Target data less than 100 examples → Feature extraction.
• 100-10000 examples → Partial fine-tuning.
• Greater than 10000 examples → Full fine-tuning.

Domain Shift Theory

Ben-David et al. (2010): Upper bound on target error:

where:

• $\epsilon_S$: source error (minimized by training).
• $d_{\mathcal{H}\Delta\mathcal{H}}$: divergence between domain representations.
• $\lambda^*$: ideal joint error (irreducible — some tasks just don't transfer).

Implication: To minimize target error, BOTH minimize source error AND minimize domain divergence in the feature space.

Unsupervised Domain Adaptation

DANN (Ganin et al., 2015): Make features domain-invariant:

Gradient reversal: Feature extractor $\theta_f$ is trained to:

1. Predict task labels well (from source).
2. CONFUSE the domain classifier (make features domain-invariant).

The domain classifier can't tell if features came from source or target → features are domain-invariant → task classifier trained on source works on target.

Other approaches:

• Maximum Mean Discrepancy (MMD): Minimize distribution distance in kernel space.
• Optimal Transport: Align source and target distributions via OT.
• Self-training: Generate pseudo-labels on target data; iterate.

Few-Shot Transfer Learning

In-context learning (LLMs): Provide a few examples in the prompt:

Input: "The food was amazing" → Positive
Input: "Terrible service" → Negative
Input: "Nice ambiance but slow" → ?

This is zero-shot transfer from pre-training to the task — no parameter updates.

Why it works: Pre-trained LLMs have learned to recognize and replicate patterns. A few examples "activate" the right behavior without changing any weights.

Limitations:

• Sensitive to example selection and ordering.
• Context window limits the number of examples.
• Doesn't match fine-tuning quality for complex tasks.

The Foundation Model Paradigm

The modern transfer learning workflow:

1. Pre-train a massive model on diverse data (expensive, done once).
2. Adapt to downstream tasks via fine-tuning, prompting, or PEFT (cheap, done many times).

Foundation models: GPT-4, LLaMA, CLIP, SAM, Whisper — each trained once, adapted to thousands of tasks.

Why it works at scale: Large models learn more transferable representations. A 70B model transfers better than a 7B model to any downstream task (even with the same fine-tuning data).

When Transfer Helps vs Hurts

Positive transfer: Source and target share relevant structure.

• ImageNet → medical imaging (shared visual features).
• English LLM → French fine-tuning (shared linguistic structure).

Negative transfer: Source domain is irrelevant or conflicting.

• ImageNet → protein structure prediction (different domain entirely).
• Formal text LLM → informal chat (style conflict).

Detection: If fine-tuning from pre-trained model performs WORSE than training from scratch, negative transfer has occurred. Solutions: use a more relevant pre-trained model or increase target data.

Common Pitfalls

Pitfall 1. Freezing ALL layers for a domain with large shift from pre-training. If the target domain is very different (e.g., X-ray images vs natural images), early layers need adaptation too.

Pitfall 2. Fine-tuning a pre-trained model on too few examples without regularization. With 50 examples and millions of parameters, overfitting is guaranteed. Use feature extraction or LoRA instead.

Pitfall 3. Assuming bigger pre-trained models always transfer better. For very narrow domains, a smaller domain-specific model can outperform a larger general-purpose model.

Summary

• Feature extraction: Freeze backbone; safe for few examples.
• Fine-tuning: Update all; best with sufficient data.
• Domain shift bound: Target error bounded by source error + domain divergence.
• DANN: Adversarial domain-invariant features for unsupervised adaptation.
• In-context learning: Zero-shot transfer via few examples in prompt.
• Foundation models: Pre-train once, adapt many times — the dominant paradigm.

Exercises

Exercise 1. For a medical imaging task with 200 labeled examples: design the transfer learning strategy (which layers to freeze, learning rate schedule).

Exercise 2. Compute the $\mathcal{H}$-divergence between two Gaussian distributions in 2D using the proxy A-distance.

Exercise 3. Compare the accuracy of: (a) training from scratch, (b) feature extraction from ImageNet, (c) full fine-tuning from ImageNet — on a dataset of 500 satellite images.

Exercise 4. Design an unsupervised domain adaptation experiment for adapting a sentiment model from product reviews (source) to movie reviews (target).

Exercise 5. When does in-context learning match fine-tuning? Estimate the crossover point (number of examples) for GPT-4 vs fine-tuned BERT on text classification.