Efficient Model Architectures

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Depthwise Separable Convolutions
5. Compound Scaling (EfficientNet)
6. Mixture of Experts (MoE)
7. Early Exit / Dynamic Depth
8. Efficient Attention Variants
9. The Accuracy-Efficiency Pareto Frontier
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Derive the FLOP reduction from depthwise separable convolutions.
2. Explain compound scaling and the relationship between depth, width, and resolution.
3. Describe MoE as conditional computation for scaling parameters cheaply.
4. Explain early exit and when dynamic depth is beneficial.
5. Navigate the accuracy-FLOPs Pareto frontier for model selection.

Notation

• $C_{\text{in}}, C_{\text{out}}$ — input/output channels
• $K$ — kernel size
• $\phi$ — compound scaling coefficient
• $E$ — number of experts, $k$ — top-k routing

Core Intuition

Standard architectures (ResNet, Transformer) are designed for accuracy with little regard for compute. Efficient architectures achieve the same accuracy with 5-10x fewer FLOPs by exploiting: factored operations (depthwise separable), conditional computation (MoE), and adaptive depth (early exit). The goal: push the Pareto frontier of accuracy vs efficiency.

Depthwise Separable Convolutions

Standard convolution: $C_{\text{in}} \times C_{\text{out}} \times K^2$ multiplications per spatial position.

Depthwise separable: Factor into two steps:

1. Depthwise: $K \times K$ convolution independently per channel. Cost: $C_{\text{in}} \times K^2$.
2. Pointwise: $1 \times 1$ convolution to mix channels. Cost: $C_{\text{in}} \times C_{\text{out}}$.

FLOP reduction:

For $C_{\text{out}}=256, K=3$: reduction = $\frac{1}{256} + \frac{1}{9} \approx 0.115$. 8.7x fewer FLOPs.

Compound Scaling (EfficientNet)

Observation: Scaling only depth, only width, or only resolution gives diminishing returns. Scaling all three together is optimal.

Compound scaling:

subject to $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (FLOP constraint).

Grid search finds $\alpha=1.2, \beta=1.1, \gamma=1.15$. Then increase $\phi$ to scale the model uniformly.

Result: EfficientNet-B7 achieves higher ImageNet accuracy than ResNet-152 with 8x fewer FLOPs.

Mixture of Experts (MoE)

Conditional computation: Only activate a subset of parameters per input:

Properties:

• Total parameters: $E \times$ dense equivalent.
• Active parameters per token: only $k/E$ fraction.
• Training compute: same as a model with $k$ experts active.

For LLMs (Mixtral 8x7B):

• 8 expert FFNs, top-2 routing.
• 47B total parameters, 13B active per token.
• Performance of a 13B dense model with cost of a 13B model.

Early Exit / Dynamic Depth

Idea: Add classification heads at intermediate layers. If the model is already confident, exit early without computing remaining layers.

Average depth: Easy examples exit early (2-3 layers); hard examples use full depth. Average compute is reduced.

For serving: Useful when latency varies by request. Easy queries answered fast; hard queries take longer.

Efficient Attention Variants

• Multi-Query Attention (MQA): Share K, V across heads → $H\times$ less KV-cache.
• Grouped-Query (GQA): $G$ groups of shared K, V → balanced tradeoff.
• Sliding Window: $O(Tw)$ instead of $O(T^2)$.
• Linear Attention: $O(Td^2)$ via kernel decomposition.
• Multi-Scale: Different heads attend at different resolutions.

The Accuracy-Efficiency Pareto Frontier

Definition: A model is Pareto-optimal if no other model achieves both higher accuracy AND lower compute.

Navigating the frontier:

• Low compute ($< 1$ GFLOP): MobileNet, EfficientNet-B0.
• Medium ($1$-$10$ GFLOP): EfficientNet-B3, DeiT-Small.
• High ($> 10$ GFLOP): Large transformers dominate.

Key insight: The Pareto frontier shifts with new architectural innovations. MoE pushed it significantly for LLMs.

Common Pitfalls

Pitfall 1. Comparing models only on FLOPs. Actual latency depends on memory access patterns, parallelism, and hardware. A model with fewer FLOPs can be slower in practice.

Pitfall 2. Using MoE without load balancing. Without auxiliary loss, some experts get all tokens (collapse) while others are unused.

Pitfall 3. Early exit with poorly calibrated confidence. If the model is overconfident at early layers, it exits too soon with wrong answers.

Summary

• Depthwise separable: 8-9x fewer FLOPs by factoring convolutions.
• Compound scaling: Scale depth, width, resolution together optimally.
• MoE: Scale parameters cheaply with conditional computation.
• Early exit: Adaptive depth — easy inputs skip expensive later layers.
• Pareto frontier: Choose architecture based on accuracy-compute budget.

Exercises

Exercise 1. Compute the FLOPs for standard vs depthwise separable convolution with $C_{\text{in}}=512, C_{\text{out}}=512, K=5, H=W=32$.

Exercise 2. For a Mixtral-style MoE with $E=8, k=2, d_{ff}=4096$: compute active vs total parameters.

Exercise 3. Design an early-exit strategy for a 32-layer transformer: where to place exit heads and what confidence threshold to use.

Exercise 4. Derive the compound scaling coefficients $\alpha, \beta, \gamma$ that double FLOPs (constraint: $\alpha\beta^2\gamma^2=2$).

Exercise 5. Compare the Pareto efficiency of a 7B dense model vs a 47B MoE (8x7B, top-2) model.