Retrieval-Augmented Generation (RAG)

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Parametric vs. Non-Parametric Memory
6. RAG Formulation
7. Dense Retrieval
8. Similarity Search and Top-K Retrieval
9. Context Construction and Generation
10. Theoretical Interpretation
11. Failure Modes
12. Worked Examples
13. Connection to the Broader Curriculum
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. Distinguish parametric (model weights) from non-parametric (retrieved docs) knowledge.
2. Formulate RAG as conditioning generation on retrieved context $z$.
3. Define dense retrieval via embedding similarity (cosine/dot product).
4. Analyze the retrieve-then-read pipeline mathematically.
5. Identify failure modes: retrieval miss, context overflow, hallucination despite retrieval.

Prerequisites

• Self-Attention — context processing
• Vectors & Linear Independence — cosine similarity
• Tokenization — context window limits

Notation

• $x$ — User query
• $y$ — Generated response
• $z, \mathcal{Z}$ — Retrieved document and corpus
• $p_\eta(z \mid x)$ — Retriever distribution
• $p_\theta(y \mid x, z)$ — Conditional generator
• $\mathcal{D} = \{d_j\}_{j=1}^M$ — Document corpus

Core Intuition

LLMs store knowledge in parameters — fixed at training time, opaque, and stale. RAG (Lewis et al., 2020) augments generation with retrieved documents from an external corpus at inference time.

Query → retrieve relevant passages → concatenate as context → LLM generates answer conditioned on context. This is non-parametric memory: the knowledge base updates without retraining.

Series context. Volume III, Chapter 16 (Applications).

Parametric vs. Non-Parametric Memory

Definition 1 (Parametric Knowledge). Facts encoded in $\theta$: $p_\theta(y \mid x)$ implicitly stores training data statistics.

Definition 2 (Non-Parametric Knowledge). External corpus $\mathcal{D} = \{d_1, \ldots, d_M\}$; retrieval selects subset at query time.

Proposition 1. Parametric memory capacity $\propto$ model size (Scaling Laws). Non-parametric memory $\propto |\mathcal{D}|$ — independently scalable.

RAG Formulation

Definition 3 (RAG Generation).

where $z$ is retrieved context, $p_\eta$ is retriever, $p_\theta$ is generator (LLM).

Practical approximation: Use top-$K$ documents with highest $p_\eta(z \mid x)$:

Dense Retrieval

Definition 4 (Bi-Encoder Retrieval).

where $E_q, E_d$ are embedding models (often shared).

Connection to Cosine Similarity: With normalized embeddings, dot product equals cosine similarity.

Definition 5 (Document Index). Precompute $\mathbf{v}_i = E_d(d_i)$ for all $d_i \in \mathcal{D}$. Store in vector index (FAISS, HNSW).

Similarity Search and Top-K Retrieval

Definition 6 (Top-K Retrieval).

Complexity. Exact search: $O(M \cdot d)$ per query. Approximate NN: $O(\log M)$ amortized.

Proposition 2. Retrieval quality upper-bounds generation quality — if $z^* \notin \mathcal{Z}_K$, model cannot ground answer in correct evidence.

Context Construction and Generation

Definition 7 (Prompt Template).

concatenated into LLM input within context window $n_{\max}$ (Tokenization limits).

Attention mechanism: Self-Attention over full context — query tokens attend to retrieved passages.

Theoretical Interpretation

Proposition 3 (Bayesian View). RAG approximates:

with discrete approximation over retrieved $z$.

Proposition 4 (Knowledge Conflict). When parametric $p_\theta$ and retrieved $z$ conflict, model may ignore retrieval — "hallucination despite RAG."

Failure Modes

1. Retrieval miss: Relevant doc not in top-$K$
2. Context truncation: $K$ docs exceed window
3. Noise: Irrelevant retrieved docs confuse generation
4. Stale index: Corpus not updated
5. Attribution failure: Model doesn't cite sources faithfully

Worked Examples

Example 1: Score Computation

$\|E_q(x)\| = \|E_d(d)\| = 1$, $\langle E_q, E_d \rangle = 0.9$ → high relevance.

Example 2: Context Budget

$K = 5$ docs × 500 tokens = 2500 tokens + query — fits in 4096 window.

Connection to the Broader Curriculum

• Self-Attention — processes retrieved context
• KV Cache — long context cost
• LoRA — fine-tune retriever or generator
• DPO — align RAG outputs

Common Pitfalls and Misconceptions

Pitfall 1: RAG does not guarantee factual accuracy.

Pitfall 2: Chunk size affects retrieval granularity.

Pitfall 3: Embedding model must match domain.

Pitfall 4: Ignoring reranking after initial retrieval.

Research Perspective

RAG (Lewis et al., 2020). Dense Passage Retrieval (Karpukhin et al., 2020). Self-RAG, corrective RAG, GraphRAG. Hybrid sparse+dense retrieval.

Summary of Takeaways

• Retriever — $p_\eta(z \mid x)$ via embeddings
• Generator — $p_\theta(y \mid x, z)$ LLM
• Index — Non-parametric memory
• Top-K — Approximate integral (6)

Next: Flash Attention (Volume IV)

Exercises

Exercise 1. Derive (1) from Bayesian conditioning.

Exercise 2. Compute retrieval score with normalized embeddings.

Exercise 3. Context window budgeting for $K$ documents.

Exercise 4. When does parametric knowledge override retrieval?

Exercise 5. Design chunking strategy for long documents.

Exercise 6. Connect retrieval to Cosine Similarity.

Exercise 7. RAG + KV Cache memory for long context.

Exercise 8. Compare RAG to fine-tuning for knowledge injection.