Data Engineering for LLM Training

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Data Pipeline
5. Web Crawling & Initial Filtering
6. Deduplication: Exact & Fuzzy
7. Quality Filtering
8. Data Mixing & Sampling
9. Tokenization
10. Data Quality vs Quantity
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Design an end-to-end data pipeline for LLM pre-training.
2. Apply MinHash deduplication at trillion-token scale.
3. Build quality classifiers for web data filtering.
4. Choose tokenization algorithms (BPE, SentencePiece, byte-level).
5. Optimize the quality-quantity tradeoff for a fixed compute budget.

Notation

• $D$ — total tokens after processing
• $q(x)$ — quality score of document $x$
• $h(x)$ — MinHash signature

Core Intuition

"Garbage in, garbage out" is amplified at scale. An LLM trained on 2T tokens of well-curated data vastly outperforms one trained on 10T tokens of unfiltered web crawl. Data engineering — crawling, filtering, deduplication, mixing — is arguably MORE important than model architecture for final quality. The best teams spend more engineering effort on data than on models.

The Data Pipeline

Stages (in order):

1. Collection: Web crawl (Common Crawl), books, code, scientific papers.
2. Language ID: Keep target language(s); remove garbage.
3. Deduplication: Remove exact and near-duplicates.
4. Quality filtering: Remove low-quality content.
5. Safety filtering: Remove toxic/harmful content.
6. Mixing: Combine domains in optimal proportions.
7. Tokenization: Convert text to token sequences.
8. Shuffling: Randomize order for training.

Scale: Common Crawl: 250B pages → after filtering: 3-5T tokens of usable text.

Web Crawling & Initial Filtering

Common Crawl: Monthly snapshots of the public web. Raw: 3.15 billion pages per month.

Initial filters (fast, rule-based):

• Remove non-text (images, video pages).
• Language detection (fastText classifier).
• Remove very short documents (fewer than 50 words).
• Remove documents with too many special characters.
• Remove known boilerplate (navigation, footers, ads).

Extraction: HTML → clean text (Trafilatura, readability).

After initial filtering: Typically 10-30% of raw crawl remains.

Deduplication: Exact & Fuzzy

Why deduplicate:

• Duplicates bias training (model memorizes repeated content).
• Waste compute on redundant data.
• Evaluation contamination (test set may be in training data).

Exact deduplication:

• Hash each document (SHA-256).
• Remove documents with identical hash.
• Fast ($O(n)$) but misses near-duplicates.

Fuzzy deduplication (MinHash + LSH):

1. Compute n-gram set for each document.
2. MinHash: generate signature of $k$ hash values ($k=128$ typical).
3. LSH (Locality Sensitive Hashing): bucket similar signatures.
4. Within each bucket: compute exact Jaccard similarity.
5. Remove documents with Jaccard greater than 0.8.

Scale: MinHash on 1B documents: $O(nk)$ compute, $O(nk)$ memory. Parallelizable.

Paragraph-level dedup: Also remove duplicate paragraphs across documents (catches boilerplate that appears in many pages).

Quality Filtering

Perplexity-based: Train a small LM on high-quality text (Wikipedia). Score web documents. Remove high-perplexity (low-quality) documents.

Classifier-based (CCNet, Gopher): Train a binary classifier:

• Positive: Wikipedia, books, curated sources.
• Negative: Random web crawl.
• Score each document; keep above threshold.

Heuristic rules (RedPajama, Dolma):

• Minimum/maximum document length.
• Maximum fraction of repeated lines.
• Maximum fraction of special characters.
• Required presence of stop words (natural language indicator).
• Alphabetical ratio (filters tables, code mixed with text).

Quality vs diversity tradeoff: Very aggressive filtering removes diverse content (dialects, informal writing, technical jargon). Some useful data looks "low quality" to classifiers trained on Wikipedia.

Data Mixing & Sampling

Typical LLM pre-training mix:

Upsampling high-quality data: Wikipedia and books may be repeated 2-5x (upsampled) because they're high quality but small volume.

Domain weighting over training: Some approaches vary the mix during training (more diverse data early, higher quality data late).

Tokenization

BPE (Byte Pair Encoding): Iteratively merge most frequent character pairs.

• Vocabulary: 32K-128K tokens.
• Handles any text (including code, multilingual).
• Standard: GPT series, LLaMA.

SentencePiece: BPE/Unigram on raw characters (language-agnostic preprocessing).

• No language-specific tokenization needed.
• Used by: T5, LLaMA, Mistral.

Byte-level BPE: Operate on UTF-8 bytes directly.

• Never produces unknown tokens.
• Handles any language or script.
• Used by: GPT-2+, Llama 3.

Vocabulary size tradeoff:

• Larger vocab: shorter sequences (faster training), more embedding parameters.
• Smaller vocab: longer sequences (slower), fewer parameters.
• Sweet spot: 32K-128K for English-centric; 128K+ for multilingual.

Data Quality vs Quantity

Key finding (Phi, Textbooks Are All You Need): Small high-quality datasets can outperform large low-quality ones:

• Phi-1 (1.3B): Trained on 7B tokens of textbook-quality code → outperforms models trained on 100B+ code tokens.

The quality-quantity frontier:

• At fixed compute: quality matters more than quantity (beyond a minimum).
• Quality improvements have diminishing returns.
• Best approach: maximum quality up to the data ceiling, then expand quantity.

Data ceiling: For English at current quality filters: approximately 5-15T high-quality tokens available. Beyond this, must lower quality threshold or use synthetic data.

Common Pitfalls

Pitfall 1. Not deduplicating against the evaluation set. If MMLU questions appear in training data, benchmarks are meaningless. Always dedup against all evaluation sets.

Pitfall 2. Over-filtering small languages. Quality classifiers trained on English Wikipedia unfairly penalize non-English content, reducing multilingual capability.

Pitfall 3. Using token count as a proxy for data quality. 1T tokens of carefully curated data teaches more than 10T tokens of unfiltered web. Focus on tokens that provide learning signal.

Summary

• Pipeline: Crawl → filter → dedup → quality → mix → tokenize.
• Deduplication: MinHash + LSH for fuzzy dedup at scale.
• Quality filtering: Perplexity scoring + classifier + heuristics.
• Mixing: 60-80% web, rest distributed across code, books, science, math.
• Tokenization: Byte-level BPE with 32-128K vocabulary.
• Quality > quantity: Curated data dramatically outperforms raw scale.

Exercises

Exercise 1. Design a MinHash dedup pipeline for 1B documents with target Jaccard threshold 0.7. Specify: n-gram size, number of hashes, LSH band configuration.

Exercise 2. Build a quality classifier: specify positive/negative training data, model architecture, and threshold selection strategy.

Exercise 3. For a 7B model trained on 1T tokens: design the optimal data mix. Justify each domain's proportion.

Exercise 4. Compare BPE with 32K vs 128K vocabulary for a multilingual model (English + Chinese + Arabic). Analyze: sequence length, embedding memory, and coverage.

Exercise 5. Estimate the total available high-quality English text on the internet (in tokens). How many more pre-training runs can this support before we exhaust the supply?