Activation Functions

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Why Nonlinearity Is Necessary
5. The Sigmoid Function
6. Hyperbolic Tangent
7. Rectified Linear Unit (ReLU)
8. Leaky ReLU and PReLU
9. GELU and Swish
10. Softmax for Classification
11. Properties Comparison
12. Common Pitfalls
13. Summary
14. Exercises

Learning Objectives

1. Prove that without nonlinearity, any depth network collapses to a single linear transformation.
2. Derive the sigmoid and its gradient; explain saturation and vanishing gradients.
3. Analyze ReLU: non-differentiability at zero, dead neurons, and unbounded activations.
4. Derive GELU as the expected value under a Gaussian mask.
5. Prove that softmax is the unique function satisfying certain axiomatic properties.

Notation

• $\sigma(z)$ — sigmoid function
• $\phi(z)$ — generic activation function
• $z = \mathbf{w}^T\mathbf{x} + b$ — pre-activation
• $\Phi(z)$ — standard Gaussian CDF
• $\phi_{\text{Gauss}}(z)$ — standard Gaussian PDF

Core Intuition

Neural networks derive their expressive power from nonlinear activation functions applied between linear transformations. Without nonlinearity, a 100-layer network computes the same function as a single matrix multiplication. The choice of activation profoundly affects gradient flow during training, the network's ability to approximate complex functions, and computational efficiency.

Why Nonlinearity Is Necessary

Theorem. A composition of affine maps is affine.

Proof. Let $f_l(\mathbf{x}) = \mathbf{W}_l\mathbf{x} + \mathbf{b}_l$ for $l = 1, \ldots, L$. Then:

which is a single affine map regardless of $L$. Adding depth without nonlinearity gains no representational power. $\blacksquare$

The Sigmoid Function

Properties:

• Range: $(0, 1)$ — interpretable as probability.
• Monotonically increasing, smooth ($C^\infty$).

Derivative:

Proof. $\sigma'(z) = \frac{e^{-z}}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}} = \sigma(z)(1-\sigma(z))$. $\blacksquare$

Maximum gradient: $\sigma'(0) = 1/4$. For $|z| > 4$: $\sigma'(z) < 0.02$.

Saturation problem: In deep networks, the gradient through $L$ sigmoid layers scales as $\prod_l \sigma'(z_l) \leq (1/4)^L$, causing exponential gradient decay (vanishing gradients).

Hyperbolic Tangent

Derivative: $\tanh'(z) = 1 - \tanh^2(z)$.

Advantages over sigmoid: Zero-centered output $\in (-1, 1)$; maximum gradient $\tanh'(0) = 1$ (vs. $1/4$ for sigmoid). Still saturates for large $|z|$.

Rectified Linear Unit (ReLU)

Derivative:

(Undefined at $z = 0$; in practice, use subgradient $\in [0,1]$.)

Advantages:

• No saturation for $z > 0$: gradient is exactly 1.
• Sparse activation: approximately 50% of neurons output zero.
• Computationally trivial: just a threshold.

Dead neuron problem: If $z \leq 0$ for all training examples (e.g., large negative bias), then $\text{ReLU}'(z) = 0$ always and the neuron receives no gradient updates. It is permanently "dead."

Leaky ReLU and PReLU

Typically $\alpha = 0.01$. Prevents dead neurons since the gradient is $\alpha \neq 0$ for negative inputs.

PReLU (Parametric ReLU): $\alpha$ is a learnable parameter, optimized during training via backpropagation.

GELU and Swish

GELU (Gaussian Error Linear Unit):

Interpretation: Multiply the input by the probability that a Gaussian random variable is less than it. This provides a smooth, probabilistic "gate."

Approximation: $\text{GELU}(z) \approx 0.5z(1 + \tanh[\sqrt{2/\pi}(z + 0.044715z^3)])$.

Derivative:

Swish: $\text{Swish}(z) = z\cdot\sigma(\beta z)$ where $\beta$ is learnable. When $\beta = 1.702$, Swish $\approx$ GELU.

Both are smooth, non-monotonic (slightly negative for $z < 0$), and have become standard in modern transformers (GPT, BERT use GELU).

Softmax for Classification

For a vector $\mathbf{z} \in \mathbb{R}^K$ (logits):

Properties:

• Output sums to 1: $\sum_k \text{softmax}(\mathbf{z})_k = 1$.
• Translation invariant: $\text{softmax}(\mathbf{z} + c\mathbf{1}) = \text{softmax}(\mathbf{z})$.
• Jacobian: $\frac{\partial \text{softmax}_i}{\partial z_j} = \text{softmax}_i(\delta_{ij} - \text{softmax}_j)$.

Temperature scaling: $\text{softmax}(\mathbf{z}/T)$ — as $T \to 0$, concentrates on $\arg\max$; as $T \to \infty$, becomes uniform.

Properties Comparison

Key properties of common activations:

• Sigmoid: smooth, bounded $(0,1)$, saturates, max gradient $= 0.25$
• Tanh: smooth, bounded $(-1,1)$, saturates, max gradient $= 1$, zero-centered
• ReLU: piecewise linear, unbounded, non-saturating for $z>0$, sparse, dead neuron risk
• Leaky ReLU: piecewise linear, unbounded, no dead neurons, non-zero everywhere
• GELU: smooth, non-monotonic, approximately linear for large $z$, standard in transformers
• Swish: smooth, non-monotonic, self-gated, interpolates between linear and ReLU

Common Pitfalls

Pitfall 1. Using sigmoid in hidden layers of deep networks. The $\leq 1/4$ gradient factor per layer causes vanishing gradients. Use ReLU or GELU instead.

Pitfall 2. Initializing weights too large with ReLU. Pre-activations become large positive → all neurons fire → no sparsity benefit. Use He initialization ($\text{Var}(w) = 2/n_{\text{in}}$).

Pitfall 3. Forgetting numerical stability in softmax. Computing $e^{z_k}$ for large $z_k$ overflows. Always subtract $\max_k z_k$ before exponentiating (doesn't change the result due to translation invariance).

Summary

• Nonlinearity is essential: without it, depth is meaningless.
• Sigmoid/tanh saturate and cause vanishing gradients in deep networks.
• ReLU solved saturation but introduced dead neurons.
• GELU/Swish are smooth, non-monotonic, and standard in modern architectures.
• Softmax converts logits to probabilities for classification.
• Activation choice directly impacts gradient flow and trainability.

Exercises

Exercise 1. Prove that $\tanh(z) = 2\sigma(2z) - 1$ and derive $\tanh'(z) = 1 - \tanh^2(z)$.

Exercise 2. Show that $\text{ReLU}(z) = \frac{1}{2}(z + |z|)$ and use this to prove it is convex.

Exercise 3. Derive the GELU derivative (equation 9) from the product rule.

Exercise 4. Prove that softmax is invariant to adding a constant: $\text{softmax}(\mathbf{z} + c\mathbf{1}) = \text{softmax}(\mathbf{z})$.

Exercise 5. For a network of depth $L$ with sigmoid activations and all pre-activations at $z=0$, compute the gradient magnitude at the first layer relative to the last. How does this scale with $L$?