Edge Deployment: Mobile, Browser & Embedded Inference

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Edge Deployment Constraints
5. Mobile Inference Frameworks
6. Browser-Based Inference
7. Model Architecture for Edge
8. On-Device Quantization & Formats
9. Power & Thermal Budgets
10. Latency Optimization Strategies
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Compare edge deployment frameworks (ONNX, TFLite, CoreML, GGUF).
2. Analyze memory/compute/power budgets for mobile and embedded targets.
3. Choose model architectures optimized for edge (MobileNet, TinyLlama).
4. Design quantization strategies for specific hardware (NPU, DSP, GPU).
5. Optimize inference latency under strict resource constraints.

Notation

• TOPS — Tera Operations Per Second (hardware capability)
• MACs — Multiply-Accumulate operations (model compute requirement)
• $P_{\text{TDP}}$ — Thermal Design Power budget

Core Intuition

Cloud inference requires network connectivity, adds latency, and costs money per request. Edge deployment runs the model ON the user's device — zero latency, full privacy, works offline. But edge devices have extreme constraints: 4GB RAM (phone), 100 GFLOPS compute (vs 1000 TFLOPS for H100), and 5W power (vs 700W for a server GPU). This requires aggressive model compression and architecture choices tailored to the hardware.

Edge Deployment Constraints

Key constraint cascade: Memory → limits model size. Compute → limits latency. Power → limits sustained throughput.

Mobile Inference Frameworks

ONNX Runtime: Cross-platform. Supports CPU, GPU, NPU backends.

• Format: .onnx (open standard).
• Optimizations: Graph optimization, quantization, operator fusion.
• Platforms: iOS, Android, Windows, Linux, Web.

TensorFlow Lite: Google's mobile framework.

• Format: .tflite (FlatBuffer).
• Quantization: Full INT8 pipeline with calibration.
• Delegates: GPU, NNAPI (Android NPU), CoreML (iOS).

CoreML: Apple's native framework.

• Format: .mlpackage.
• Targets: Neural Engine (NPU), GPU, CPU.
• Advantage: Best performance on Apple devices.

llama.cpp / GGUF: LLM-specific inference.

• Format: .gguf (quantized model format).
• Quantization: Q2_K through Q8_0 (2-8 bit).
• Platforms: Any CPU (AVX2, ARM NEON); Metal GPU on Mac.
• Use case: Running 7B models on laptops/phones.

Browser-Based Inference

WebGPU: Next-gen browser GPU API.

• Access to GPU compute from JavaScript.
• Comparable to native Vulkan/Metal performance.
• Libraries: WebLLM, Transformers.js.

WebAssembly (WASM): CPU inference in browser.

• Portable, secure, sandboxed.
• 2-5x slower than native CPU (improving).
• SIMD support for vector operations.

Practical for:

• Small models (less than 1B parameters quantized).
• Privacy-critical applications (data never leaves browser).
• Offline web apps.

Limitations: No persistent memory across page loads. Limited to device GPU memory. WebGPU still not universal.

Model Architecture for Edge

MobileNet (depthwise separable convolutions):

• Standard conv: $d_k^2 \times c_{\text{in}} \times c_{\text{out}}$ MACs.
• Depthwise separable: $d_k^2 \times c_{\text{in}} + c_{\text{in}} \times c_{\text{out}}$ MACs.
• Reduction: $\sim 8$-$9\times$ fewer MACs.

EfficientNet: NAS-found architecture; Pareto-optimal accuracy vs FLOPs.

TinyLlama / SmolLM: Small LLMs designed for edge:

• 1-3B parameters.
• Trained on more data (over-trained by Chinchilla standards).
• INT4 quantized: fits in 1-2GB RAM.

Architecture principles for edge:

• Prefer depth over width (more layers, smaller hidden dim).
• Use grouped convolutions / GQA (reduce memory bandwidth).
• Avoid operations poorly supported by NPUs (dynamic shapes, custom ops).

On-Device Quantization & Formats

GGUF quantization levels:

NPU-friendly quantization:

• INT8 symmetric (most NPUs natively support).
• Per-channel calibration for maximum accuracy.
• Avoid mixed-precision (NPUs prefer uniform precision).

Power & Thermal Budgets

Power consumption model:

where $C$ is capacitance, $V$ is voltage, $f$ is frequency.

Practical implications:

• Sustained inference throttles frequency (thermal limits).
• Burst inference: full speed for seconds, then throttle.
• Battery impact: 1W continuous = 20% battery drain per hour on phone.

Optimization: Minimize total operations per inference. A 2x faster model saves 2x energy (same result, shorter active time).

Latency Optimization Strategies

1. Batch size = 1 optimization:

• On edge, always batch=1 (single user).
• Optimize for memory bandwidth (not compute).
• Quantize weights aggressively (fewer bytes to load).

2. Operator fusion (critical on mobile):

• Mobile GPUs have high kernel launch overhead.
• Fuse Conv+BN+ReLU, Attention operations, etc.
• TFLite and CoreML do this automatically.

3. Speculative decoding on edge:

• Use tiny draft model (100M) for a 3B target.
• 1.5-2x speedup on mobile.

4. Caching & preprocessing:

• Cache system prompt KV (don't recompute).
• Pre-tokenize common inputs.

Common Pitfalls

Pitfall 1. Testing on desktop GPU, deploying on mobile NPU. Performance characteristics are completely different. A model fast on RTX 4090 may be slow on Snapdragon NPU (different supported operations, memory hierarchy).

Pitfall 2. Ignoring model loading time. On mobile, loading a 4GB model from flash takes 2-5 seconds. First-inference latency includes this. Solutions: lazy loading, memory mapping.

Pitfall 3. Dynamic shapes on NPUs. Most NPUs compile models for FIXED shapes. Dynamic sequence lengths require recompilation or padding. Use fixed bucket sizes (128, 256, 512, 1024).

Summary

• Edge constraints: 4-8GB RAM, 15 TOPS compute, 5W power.
• Frameworks: ONNX (cross-platform), TFLite (mobile), CoreML (Apple), GGUF (LLMs).
• Browser: WebGPU for GPU access; limited to smaller models.
• Architecture: Depthwise separable convs, GQA, small-but-overtrained LLMs.
• Quantization: GGUF Q4-Q8; NPU-friendly INT8.
• Power: Minimize operations = minimize energy. Thermal throttling limits sustained performance.

Exercises

Exercise 1. For a 3B model in Q4_K_M (1.7GB): compute the time to generate 100 tokens on iPhone 15 Pro Neural Engine (17 TOPS INT8, 100GB/s bandwidth).

Exercise 2. Compare the latency of: (a) cloud inference via API (100ms network + 50ms compute), (b) on-device inference (500ms compute, 0ms network). When does on-device win?

Exercise 3. Design an edge deployment pipeline for a summarization model: specify model choice, quantization, framework, and target device.

Exercise 4. Compute the battery drain of running continuous on-device LLM inference at 5 tokens/second on a 4500mAh phone battery.

Exercise 5. Compare WebGPU vs native (llama.cpp Metal) performance for Llama-3 8B Q4 on an M3 MacBook. Estimate the overhead of the browser abstraction layer.