ControlNet & Spatial Conditioning

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Problem: Spatial Control
5. ControlNet Architecture
6. Zero Convolution
7. Training ControlNet
8. IP-Adapter: Image Conditioning
9. Multi-Conditioning
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Explain why text prompts alone are insufficient for precise spatial control.
2. Derive the ControlNet architecture and the role of zero-convolution.
3. Prove that zero-convolution preserves the pre-trained model at initialization.
4. Describe IP-Adapter's mechanism for image-based conditioning.
5. Explain how multiple conditions can be composed.

Notation

• $\mathbf{c}_s$ — spatial condition (edge map, depth map, pose, etc.)
• $\mathcal{F}_\theta$ — frozen U-Net copy (ControlNet branch)
• $\mathcal{Z}(\cdot; \mathbf{0})$ — zero-convolution (initialized to zero)

Core Intuition

Text prompts specify what to generate but not where or how. "A person standing" doesn't tell the model the person's pose, the layout, or the depth structure. ControlNet adds a parallel branch that processes spatial signals (edges, depth, pose) and injects spatial guidance into the denoising process — without retraining the base model.

The Problem: Spatial Control

Text-only control limitations:

• "A house with a red roof" → correct semantics, random layout.
• Cannot specify: exact edge structure, depth ordering, human pose, segmentation layout.

Need: Condition on spatial maps $\mathbf{c}_s \in \mathbb{R}^{H \times W \times C_s}$ (Canny edges, OpenPose skeleton, depth map, etc.).

ControlNet Architecture

Key design: Clone the encoder of the pre-trained U-Net; the clone processes the spatial condition. Its outputs are added to the original U-Net's decoder via zero-convolutions.

where $\mathcal{Z}(\cdot; \mathbf{0})$ is a $1 \times 1$ convolution initialized to zero weights and zero bias.

Structure:

• Locked (frozen) U-Net: Original pre-trained model, unchanged.
• Trainable copy: Processes $\mathbf{x}_t$ augmented with spatial condition.
• Zero-conv connections: Bridge between copy and original.

Zero Convolution

Definition: A $1 \times 1$ convolution with weights and biases initialized to zero:

Critical property at initialization: Since the output is zero, the ControlNet branch contributes nothing to the frozen U-Net. The model starts as the original pre-trained model.

During training: Weights gradually become non-zero, smoothly introducing the control signal. This prevents the catastrophic corruption that direct fine-tuning would cause.

Gradient flow: Despite zero forward output, gradients still flow through the trainable copy because:

Training ControlNet

Data: Pairs of (image, spatial condition) — e.g., (image, Canny edge map of image).

Objective: Same as diffusion training, but conditioned on spatial signal:

Only the ControlNet branch trains. The original U-Net stays frozen.

Training cost: ~50% of training a full model (only the encoder copy + zero-convs are trained).

IP-Adapter: Image Conditioning

Problem: Condition on a reference image (style, subject) rather than spatial layout.

Approach: Extract image features via CLIP image encoder, then inject via decoupled cross-attention:

where $\mathbf{c}_{\text{image}} = \text{Proj}(\text{CLIP}_{\text{image}}(\mathbf{I}_{\text{ref}}))$ and $\lambda$ controls the influence strength.

Advantage: Only trains the image cross-attention layers (few parameters). Compatible with ControlNet simultaneously.

Multi-Conditioning

Composition: Multiple ControlNets can be applied simultaneously:

Example: Canny edge ControlNet + depth ControlNet + text prompt → precise control over both structure and depth.

Weight balancing: $w_i$ controls the strength of each condition. Higher $w$ = stronger adherence to that condition (at the cost of text alignment).

Common Pitfalls

Pitfall 1. Training ControlNet with misaligned condition-image pairs. If the edge map doesn't correspond to the actual image, the model learns noise.

Pitfall 2. Setting control weight too high. The model may ignore the text prompt entirely and produce an image that rigidly follows the spatial condition.

Pitfall 3. Applying ControlNet trained at one resolution to a different resolution. Spatial conditions are resolution-dependent; mismatched resolution degrades quality.

Summary

• ControlNet adds spatial conditioning without modifying the pre-trained model.
• Zero-convolution ensures the model starts identical to the base model.
• Trainable encoder copy processes spatial signals (edges, depth, pose).
• IP-Adapter enables image-based conditioning via decoupled cross-attention.
• Multiple conditions can be composed with weighted summation.

Exercises

Exercise 1. Prove that at initialization, a ControlNet-augmented model produces identical outputs to the base model.

Exercise 2. Compute the additional parameters introduced by ControlNet for a U-Net with 860M encoder parameters.

Exercise 3. Explain why zero-convolution is preferred over simply initializing the copy with the pre-trained weights and fine-tuning directly.

Exercise 4. Design a training pipeline for a ControlNet that conditions on semantic segmentation maps.

Exercise 5. Derive the gradient of the loss with respect to the zero-convolution weights at the first training step.