Learning Objectives

Distinguish unstructured from structured pruning and explain hardware implications.
Apply Taylor expansion importance scoring for channel selection.
Identify redundant attention heads and layers in transformers.
Design iterative pruning schedules with recovery fine-tuning.
Map structured sparsity patterns to hardware acceleration.

Notation

$\mathbf{W}^{(l)} \in \mathbb{R}^{c_{\text{out}} \times c_{\text{in}} \times k \times k}$ — convolutional filter at layer $l$
$\mathcal{I}(s)$ — importance score of structure $s$
$\Delta\mathcal{L}$ — change in loss from removing a structure

Core Intuition

Unstructured pruning (zeroing individual weights) achieves high compression but provides no actual speedup on standard hardware — the tensor still has the same shape, just with zeros. Structured pruning removes ENTIRE channels, heads, or layers — producing genuinely smaller models that run faster WITHOUT special sparse hardware. The challenge: structured removal is more aggressive and requires careful importance estimation.

Structured Pruning

Prune %

0.30

Kept channelsPruned rows

Explore: Structured pruning removes entire channels, heads, or layers — unlike unstructured pruning, it achieves real speedups on standard hardware without sparse kernels.

Unstructured vs Structured Pruning

Unstructured (weight-level):

Remove individual weights: $W_{ij} = 0$ .
High sparsity achievable (90%+).
No speedup on GPUs without sparse hardware.
Requires sparse matrix formats (CSR, CSC) for any benefit.

Structured (unit-level):

Remove entire channels, heads, layers, or rows.
Lower sparsity achievable (50-70% typical).
DIRECT speedup on all hardware (smaller dense matrices).
Produces a standard dense model (no sparse formats needed).

N:M sparsity (semi-structured):

Keep $N$ values per block of $M$ (e.g., 2:4 = 50% sparsity).
Supported natively by NVIDIA A100/H100 Tensor Cores.
2x speedup with only 50% sparsity.

Channel Pruning in CNNs

Remove entire output channels (filters) from convolutional layers:

If channel $j$ in layer $l$ is pruned:

Remove filter $\mathbf{W}^{(l)}[j, :, :, :]$ entirely.
Remove corresponding input channel in layer $l+1$ : $\mathbf{W}^{(l+1)}[:, j, :, :]$ .
Remove bias $b_j^{(l)}$ and BN parameters for channel $j$ .

FLOP reduction: Removing fraction $p$ of channels reduces FLOPs by approximately $p^2$ (affects both the pruned layer and next layer).

Importance criteria for channels:

L1-norm of filter: $\mathcal{I}_j = \|\mathbf{W}^{(l)}[j]\|_1$ (Li et al., 2017).
Batch normalization scaling: $\mathcal{I}_j = |\gamma_j|$ (channels with small BN scale contribute little).
Activation magnitude: $\mathcal{I}_j = \mathbb{E}[\|a_j\|]$ (rarely-activated channels are unimportant).

Attention Head Pruning

Transformers have redundant heads. Michel et al. (2019) showed 20-40% of heads can be removed with minimal accuracy loss.

Head importance score:

\mathcal{I}_h = \mathbb{E}_{x}\left|\frac{\partial\mathcal{L}}{\partial\xi_h}\right|, \quad \text{where } \xi_h \text{ gates head } h. \tag{1}

Alternative: confidence-based pruning.

Compute attention entropy per head: $H_h = -\sum_j \alpha_j \log\alpha_j$ .
Heads with very low entropy (attend to one token always) may be redundant.
Heads with very high entropy (uniform attention) contribute little signal.

Width pruning (neurons within FFN):

Remove rows from $\mathbf{W}_1$ and corresponding columns from $\mathbf{W}_2$ in the FFN.
Reduce FFN dimension from $4d$ to $rd$ where $r < 4$ .

Layer Pruning & Depth Reduction

Some transformer layers contribute minimally (especially middle layers).

Layer importance metrics:

Cosine similarity: $\text{sim}(\mathbf{x}^{(l)}, \mathbf{x}^{(l+1)})$ . If input ≈ output, layer does nothing.
Block influence: $\mathcal{I}_l = \|\mathbf{x}^{(l+1)} - \mathbf{x}^{(l)}\|^2 / \|\mathbf{x}^{(l)}\|^2$ .
Fisher information: Sensitivity of loss to removing the layer.

Findings (LLMs):

First and last layers are critical (cannot be removed).
Middle layers (layers 10-20 in a 32-layer model) often have high redundancy.
Removing 25% of layers with fine-tuning recovers within 1-2% accuracy.

ShortGPT (Men et al., 2024): Remove layers with lowest "Block Influence" score. LLaMA-2 70B → 55 layers (from 80) with minimal degradation.

Importance Scoring Methods

1. Magnitude-based: $\mathcal{I}(s) = \|s\|_p$ (L1 or L2 norm of weights in structure).

Simple, fast.
Limitation: small weights may still be important (compensatory).

2. Taylor expansion (first-order):

\mathcal{I}(s) = \left|\Delta\mathcal{L}\right| \approx \left|\mathbf{g}_s^T \mathbf{w}_s\right|, \tag{2}

where $\mathbf{g}_s$ is gradient and $\mathbf{w}_s$ is weight vector of structure $s$ .

3. Taylor expansion (second-order, OBS):

\mathcal{I}(s) = \frac{w_s^2}{2[H^{-1}]_{ss}}, \tag{3}

using the inverse Hessian diagonal. More accurate but computationally expensive.

4. Sensitivity analysis: Temporarily remove structure, measure loss increase:

\mathcal{I}(s) = \mathcal{L}(\theta \setminus s) - \mathcal{L}(\theta). \tag{4}

Most accurate but requires $O(S)$ forward passes.

Iterative Pruning & Fine-tuning

One-shot pruning: Remove all structures at once → large accuracy drop.

Iterative (gradual) pruning:

Prune a small fraction (10-20%).
Fine-tune for recovery (few epochs).
Repeat until target sparsity.

Schedule (cubic):

s_t = s_f + (s_i - s_f)\left(1 - \frac{t - t_0}{n\Delta t}\right)^3, \tag{5}

where $s_t$ is sparsity at step $t$ , $s_i$ is initial, $s_f$ is final target.

Lottery Ticket Hypothesis (Frankle & Carlin, 2019): Within a large network, there exists a small subnetwork that, when trained from initialization, matches the full network's performance. Iterative magnitude pruning finds these "winning tickets."

Hardware Acceleration of Structured Sparsity

2:4 sparsity on NVIDIA Tensor Cores:

For every 4 consecutive values, 2 must be zero.
Stored as: 2 values + 2-bit indices (75% storage, 2x compute speed).
Supported: A100, H100, Blackwell architecture.

Block sparsity: Zero out entire blocks (e.g., 32×32 or 64×64).

Allows standard dense GEMM on non-zero blocks.
4-8x speedup at 75-87.5% sparsity.

Channel pruning → standard dense ops:

After removing channels, model is just a smaller dense model.
Runs on any hardware without modification.
Best portability.

Common Pitfalls

Pitfall 1. Pruning by magnitude alone in transformers. In attention mechanisms, small weights can be critical (they prevent attention to irrelevant tokens). Taylor-based importance is much more reliable.

Pitfall 2. Pruning without fine-tuning recovery. Even careful pruning causes accuracy loss. Always budget fine-tuning steps proportional to pruning aggressiveness.

Pitfall 3. Expecting 90% structured sparsity to work. Unlike unstructured pruning (90%+ achievable), structured pruning typically maxes out at 50-70% before severe degradation. Be realistic about targets.

Summary

Structured pruning removes entire channels/heads/layers → direct speedup.
Channel pruning: L1 norm, BN scaling, or activation-based importance.
Head pruning: 20-40% of transformer heads removable with minimal loss.
Layer pruning: Middle layers often redundant; 25% removable.
Taylor expansion provides principled importance scoring.
Iterative pruning with fine-tuning achieves better accuracy than one-shot.
2:4 sparsity offers 2x speedup on modern NVIDIA hardware.

Exercises

Exercise 1. For a ResNet-50: compute the FLOP reduction from removing 30% of channels in each layer (accounting for inter-layer dependencies).

Exercise 2. Implement Taylor first-order importance scoring for attention heads in a 12-head transformer. Which heads are least important?

Exercise 3. Compare one-shot vs iterative pruning (5 rounds) for removing 50% of parameters from BERT. Plot accuracy vs pruning fraction.

Exercise 4. Design a 2:4 sparsity training recipe for a GPT-2 model: specify when to introduce sparsity, how to maintain it during training, and expected speedup.

Exercise 5. For LLaMA-7B with 32 layers: compute the Block Influence score and determine the optimal set of 8 layers to remove.

Structured Pruning: Channels, Heads & Layers

Table of Contents