Dark Knowledge & Information in Soft Labels

Learning Objectives
Notation
Core Intuition
Hard Labels vs Soft Labels: Information Content
Dark Knowledge in Wrong Classes
Temperature and Information Release
Inter-Class Similarity Structure
The Gradient Perspective
Label Smoothing as Primitive Distillation
Information-Theoretic Analysis
Common Pitfalls
Summary
Exercises

Learning Objectives

Quantify the information content of soft vs hard labels.
Explain "dark knowledge" — what the teacher knows beyond correct/incorrect.
Derive why temperature scaling reveals hidden structure.
Connect label smoothing to a uniform-teacher distillation.
Analyze the gradient signal from soft vs hard labels.

Notation

$K$ — number of classes
$p_i^T$ — teacher probability for class $i$ at temperature $T$
$H(p)$ — entropy of distribution $p$
$I(X; Y)$ — mutual information

Core Intuition

A hard label "cat" gives 1 bit of information (which of $K$ classes). A soft label $[0.7, 0.2, 0.05, 0.03, 0.02]$ gives information about EVERY class simultaneously: the image is mostly cat, somewhat dog, slightly fox, unlikely car, very unlikely airplane. This rich signal tells the student about the teacher's learned similarity structure — which classes are confusable, how confident to be, and what features matter.

Dark Knowledge in Soft Targets

Temp T

4.00

Hard labelsSoft targets

Explore: Soft targets at high T reveal dark knowledge — relative similarities between classes (e.g., dog is more cat-like than fish). This is what distillation transfers beyond hard labels.

Hard Labels vs Soft Labels: Information Content

Hard label (one-hot): $y = [0, 0, 1, 0, \ldots, 0]$

Information content: $\log_2(K)$ bits (identifies one class out of $K$ ).

Soft label: $p = [p_1, p_2, \ldots, p_K]$

Information content per sample: up to $(K-1) \times$ precision bits. With FP16 probabilities over $K=1000$ classes: potentially $1000 \times 16 = 16000$ bits.

Effective information: Not all bits are useful (many classes have near-zero probability), but the effective information is:

I_{\text{effective}} \approx H(p) \cdot \text{precision}_{\text{bits}} \gg \log_2(K). \tag{1}

For a soft distribution with entropy $H(p) = 3$ nats over 1000 classes: roughly 10-20x more information than a hard label.

Dark Knowledge in Wrong Classes

Hinton's key insight (2015): The probabilities assigned to INCORRECT classes are the real treasure.

Example — image of "2" in MNIST:

Hard label: "2" (1 bit of information).
Soft label: P(2)=0.8, P(3)=0.1, P(7)=0.05, P(8)=0.03, P(0)=0.02.

What the student learns:

"2" looks somewhat like "3" (both have curves).
"2" resembles "7" (the top stroke).
"2" is slightly like "8" (the loops).
"2" is nothing like "0", "1", "4", "5", "6", "9".

This structural information about inter-digit similarity is invisible in hard labels but crucial for generalization.

Temperature and Information Release

At standard temperature ( $T=1$ ), the teacher's softmax is nearly one-hot (confident trained model):

p_i^{T=1} = \frac{e^{z_i}}{\sum_j e^{z_j}} \approx \begin{cases} 0.99 & i = \text{correct} \\ 0.001 & \text{others} \end{cases} \tag{2}

The dark knowledge is hidden in the tiny probabilities (numerically indistinguishable from zero).

At high temperature ( $T=4$ ):

p_i^T = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}} \approx \begin{cases} 0.5 & i = \text{correct} \\ 0.1, 0.05, \ldots & \text{others} \end{cases} \tag{3}

The inter-class structure becomes visible. Dark knowledge is "released."

Information scaling with temperature: As $T$ increases from 1:

Entropy of soft labels increases ( $H(p^T) \to \log K$ as $T \to \infty$ ).
Initially: MORE useful information (structure revealed).
Eventually: LESS useful (approaches uniform — no signal).
Optimal $T$ : Where structure is visible but signal is still present ( $T = 2$ – $20$ ).

Inter-Class Similarity Structure

The soft label encodes a similarity metric learned by the teacher:

\text{sim}(i, j) \propto p_j^T \quad \text{for samples of class } i. \tag{4}

Averaging over all samples of class $i$ :

\mathbf{S}_{ij} = \mathbb{E}_{x \sim \text{class}_i}[p_j^T(x)] \tag{5}

forms a similarity matrix. This matrix reveals:

Which classes share features (high off-diagonal).
Which classes are completely unrelated (near-zero).
Hierarchical structure (e.g., "dog breeds" cluster together).

The student learns this metric implicitly — without ever being told class relationships.

The Gradient Perspective

Hard label gradient (cross-entropy):

\frac{\partial\mathcal{L}_{\text{hard}}}{\partial z_i} = q_i - y_i = \begin{cases} q_i - 1 & i = \text{correct} \\ q_i & i \neq \text{correct} \end{cases} \tag{6}

Only the correct class gets a "pull toward 1" signal. Wrong classes just get "push toward 0."

Soft label gradient (KL with teacher):

\frac{\partial\mathcal{L}_{\text{soft}}}{\partial z_i} = \frac{1}{T}(q_i^T - p_i^T) \tag{7}

EVERY class gets a gradient proportional to (student - teacher) probability. Classes where the student disagrees with the teacher get the strongest signal, regardless of whether they're correct.

Richer signal: With $K=1000$ classes, hard labels give 1 useful gradient per sample. Soft labels give 1000 gradients — every class provides a learning signal.

Label Smoothing as Primitive Distillation

Label smoothing (Szegedy et al., 2016): Replace hard labels with:

y_i^{\text{smooth}} = (1 - \epsilon) \cdot y_i + \frac{\epsilon}{K}. \tag{8}

This is distillation from a trivial "teacher" that predicts uniform probability for all wrong classes.

Why it helps: Prevents overconfidence (same as distillation's regularization). But it provides NO structural information (all wrong classes treated equally).

Distillation is MUCH better: The teacher provides DIFFERENTIATED probabilities for wrong classes — "dog" gets 0.1, "car" gets 0.001. Label smoothing gives both $\epsilon/K$ .

Information-Theoretic Analysis

Bits of information per training example:

Method	Bits per example	Source
Hard label	$\log_2(K)$	Class identity
Label smoothing	$\log_2(K) + 1$	Class + slight uniformity
Soft labels ( $T=4$ )	$\sim K \cdot H(p)/\log 2$	Full distribution
Feature distillation	$d \times$ precision	Hidden layer vector

For ImageNet ( $K=1000$ ):

Hard: 10 bits.
Soft ( $T=4$ ): 50-100 effective bits.
Feature (d=2048, FP16): 32768 bits.

Implication: Distillation is equivalent to training with 5-10x more labeled data (in terms of information per sample).

Common Pitfalls

Pitfall 1. Thinking dark knowledge only matters for classification. In generation tasks (LLMs), the token-level distribution contains information about style, uncertainty, and alternatives — all are "dark knowledge."

Pitfall 2. Using too low temperature ( $T=1$ ) and expecting distillation to help. At $T=1$ , the teacher is nearly one-hot — barely more informative than a hard label.

Pitfall 3. Assuming all wrong-class probabilities are noise. The teacher's wrong-class probabilities reflect genuine uncertainty and learned similarity — they're signal, not noise.

Summary

Dark knowledge: Information in the teacher's incorrect-class probabilities.
Quantification: Soft labels provide 5-10x more information per sample than hard labels.
Temperature releases hidden structure: too low = one-hot; too high = uniform.
Gradient signal: Every class provides learning signal (not just the correct one).
Inter-class similarity is implicitly encoded in soft distributions.
Label smoothing is primitive distillation; real distillation is far richer.

Exercises

Exercise 1. For a 10-class problem with teacher logits $[8, 5, 3, 2, 1, 0, -1, -2, -3, -5]$ : compute soft labels at $T=1, 4, 10$ . At which temperature is the inter-class structure most visible?

Exercise 2. Prove that the KL divergence between soft labels and hard labels decreases with temperature (more similar at high $T$ ).

Exercise 3. Compute the entropy (in bits) of the soft label distribution at $T=1$ and $T=4$ for the logits in Exercise 1.

Exercise 4. Construct the inter-class similarity matrix $\mathbf{S}$ for CIFAR-10 using a pre-trained ResNet's soft predictions. Which classes are most similar?

Exercise 5. Derive the effective "data multiplier" of distillation: how many hard-label examples would provide the same information as one soft-label example?