Classification Metrics

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Confusion Matrix
5. Precision, Recall, and F-Score
6. The Precision-Recall Tradeoff
7. ROC Curve and AUC
8. Calibration
9. Multi-Class Extensions
10. Proper Scoring Rules
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Define and compute all confusion matrix-derived metrics.
2. Derive the relationship between classification threshold and precision/recall.
3. Prove that AUC equals the probability a random positive ranks above a random negative.
4. Define calibration and explain reliability diagrams.
5. Distinguish proper from improper scoring rules.
6. Choose appropriate metrics for different problem settings (imbalanced classes, ranking, probability estimation).

Notation

• $\text{TP}, \text{FP}, \text{TN}, \text{FN}$ — true/false positives/negatives
• $P = \text{TP} + \text{FN}$ — total positives
• $N = \text{TN} + \text{FP}$ — total negatives
• $\hat{p}(x) \in [0,1]$ — predicted probability of positive class
• $\tau \in [0,1]$ — classification threshold

Core Intuition

Accuracy alone is misleading for imbalanced datasets. A spam filter that labels everything "not spam" achieves 99% accuracy if only 1% of emails are spam, yet catches nothing. We need metrics that separately measure performance on positive and negative classes, and that evaluate the quality of probability estimates, not just hard decisions.

The Confusion Matrix

For a binary classifier with threshold $\tau$ (predict positive if $\hat{p}(x) \geq \tau$):

Precision, Recall, and F-Score

Precision (positive predictive value):

Recall (sensitivity, true positive rate):

F$_\beta$-Score (harmonic mean weighted by $\beta$):

$F_1$ ($\beta = 1$) weights precision and recall equally. $\beta > 1$ emphasizes recall; $\beta < 1$ emphasizes precision.

The Precision-Recall Tradeoff

As threshold $\tau$ decreases:

• More samples classified as positive → TP increases, FP increases.
• Recall increases (we catch more positives).
• Precision may decrease (more false positives among predictions).

The precision-recall curve plots precision vs. recall as $\tau$ varies from 1 to 0. The area under the PR curve (AP) summarizes performance across all thresholds.

ROC Curve and AUC

The ROC curve plots True Positive Rate vs. False Positive Rate:

Theorem. The AUC (Area Under ROC Curve) equals the probability that a randomly chosen positive example has a higher predicted score than a randomly chosen negative:

where $X^+ \sim P_{X|Y=1}$ and $X^- \sim P_{X|Y=0}$.

Proof sketch. The AUC can be written as $\int_0^1 \text{TPR}(\text{FPR}^{-1}(t))\,dt$. By a change of variables using the score distributions of positives and negatives, this equals the Wilcoxon-Mann-Whitney U-statistic, which computes pairwise comparison probabilities. $\blacksquare$

Interpretation: AUC = 0.5 is random guessing; AUC = 1.0 is perfect separation.

Calibration

A classifier is calibrated if:

Reliability diagram: Bin predictions by $\hat{p}$, plot observed frequency vs. predicted probability. A perfectly calibrated model lies on the diagonal.

Expected Calibration Error (ECE):

where $S_b$ is the set of samples in bin $b$, $\text{acc}$ is accuracy, and $\text{conf}$ is average predicted probability.

Platt scaling: Post-hoc calibration by fitting $P(Y=1 \mid f(x)) = \sigma(af(x) + b)$ on a validation set.

Multi-Class Extensions

For $K$ classes, metrics are aggregated:

• Macro-average: Compute metric per class, then average. Treats all classes equally.
• Micro-average: Aggregate TP, FP, FN across classes, then compute metric. Weighted by class frequency.
• Weighted average: Average per-class metric weighted by support (class size).

Multi-class AUC: One-vs-rest AUC averaged across classes, or pairwise AUC averaged across all $(K \choose 2)$ pairs.

Proper Scoring Rules

Definition. A scoring rule $S(p, y)$ is proper if the expected score is maximized (or minimized, depending on convention) when the predicted distribution equals the true distribution.

• Log loss (cross-entropy): proper. $S(p, y) = -y\log p - (1-y)\log(1-p)$.
• Brier score: proper. $S(p, y) = (p - y)^2$.
• Accuracy at threshold 0.5: improper (does not incentivize calibrated probabilities).

Theorem. A proper scoring rule incentivizes honest probability reporting: if the true probability is $q$, the forecaster minimizes expected loss by reporting $\hat{p} = q$.

Common Pitfalls

Pitfall 1. Using accuracy on imbalanced data. A 1% positive rate means 99% accuracy by always predicting negative. Use PR-AUC or F1 instead.

Pitfall 2. Comparing AUC across different datasets. AUC depends on the negative class distribution; a model with 0.95 AUC on one task isn't necessarily better than 0.90 on another.

Pitfall 3. Optimizing for F1 during training. F1 is not differentiable and threshold-dependent. Train with a proper scoring rule (log loss), then optimize the threshold for F1 post-hoc.

Summary

• The confusion matrix encodes all binary classification outcomes at a given threshold.
• Precision and recall measure complementary aspects; $F_1$ balances them.
• ROC-AUC measures ranking quality; equals the pairwise comparison probability.
• Calibration measures probability quality; assessed via reliability diagrams and ECE.
• Proper scoring rules (log loss, Brier) incentivize honest probability estimates.
• Choose metrics based on the task: ranking → AUC; imbalanced detection → PR-AUC/F1; probability estimation → log loss + calibration.

Exercises

Exercise 1. Prove that $F_1 = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}$.

Exercise 2. Show that AUC = 0.5 for a random scorer and AUC = 1 for a perfect separator.

Exercise 3. Prove that log loss is a proper scoring rule: the expected log loss $\mathbb{E}_Y[-Y\log p - (1-Y)\log(1-p)]$ is minimized at $p = P(Y=1)$.

Exercise 4. For a dataset with 100 positives and 10000 negatives, compute the precision of a classifier with TPR = 0.9 and FPR = 0.01.

Exercise 5. Derive the Brier score decomposition: $\text{BS} = \text{reliability} - \text{resolution} + \text{uncertainty}$.