Types of Transformer Architectures

Learning Objectives
Notation
Core Intuition
Encoder-Only (BERT-style)
Decoder-Only (GPT-style)
Encoder-Decoder (T5-style)
Prefix LM (UniLM-style)
Mixture of Experts (MoE)
State Space Models (Mamba)
Architecture Comparison
Common Pitfalls
Summary
Exercises

Learning Objectives

Formalize the attention mask patterns for each architecture type.
Derive the training objectives (MLM, CLM, span corruption) for each.
Explain why decoder-only models dominate scaling.
Describe MoE as a sparse conditional computation strategy.
Explain the recurrence in state-space models and their linear complexity.

Notation

$\mathbf{M}_{\text{causal}} \in \{0, -\infty\}^{T \times T}$ — causal mask (lower triangular)
$\mathbf{M}_{\text{bidir}}$ — bidirectional mask (all zeros, no masking)
$p_\theta(x_t \mid x_{<t})$ — causal language model
$p_\theta(x_{\text{mask}} \mid x_{\text{visible}})$ — masked language model

Core Intuition

The three main transformer variants differ in what each position can attend to:

Encoder-only (bidirectional): every position sees everything — best for understanding/classification.
Decoder-only (causal): each position sees only the past — best for generation.
Encoder-decoder: encoder is bidirectional; decoder is causal but cross-attends to encoder — best for sequence-to-sequence tasks.

Transformer Architectures

Layers

Explore: Encoder-only uses bidirectional attention for understanding. Decoder-only is causal for generation. Encoder-decoder adds cross-attention for seq2seq tasks.

Encoder-Only (BERT-style)

Architecture: $L$ transformer layers with bidirectional self-attention (no masking).

Attention pattern: Full attention matrix — position $i$ attends to all positions $j \in \{1, \ldots, T\}$ :

\mathbf{A} = \text{softmax}\left(\frac{\mathbf{QK}^T}{\sqrt{d_k}}\right). \quad \text{(no mask)} \tag{1}

Training objective: Masked Language Modeling (MLM). Randomly mask 15% of tokens; predict the masked tokens from context:

\mathcal{L}_{\text{MLM}} = -\sum_{t \in \text{masked}} \log p_\theta(x_t \mid \mathbf{x}_{\backslash t}). \tag{2}

Strengths: Rich bidirectional representations; excellent for classification, NER, sentence similarity.

Limitations: Cannot generate text autoregressively (no natural left-to-right ordering).

Models: BERT, RoBERTa, DeBERTa, ELECTRA.

Decoder-Only (GPT-style)

Architecture: $L$ transformer layers with causal self-attention.

Attention pattern: Lower-triangular mask — position $i$ attends only to $j \leq i$ :

\mathbf{A} = \text{softmax}\left(\frac{\mathbf{QK}^T}{\sqrt{d_k}} + \mathbf{M}_{\text{causal}}\right). \tag{3}

Training objective: Causal Language Modeling (CLM). Predict the next token:

\mathcal{L}_{\text{CLM}} = -\sum_{t=1}^T \log p_\theta(x_t \mid x_1, \ldots, x_{t-1}). \tag{4}

Strengths: Natural for generation; simple training (next-token prediction); emergent in-context learning at scale; most scalable architecture.

Why it dominates: One unified objective for all tasks (framed as text completion). KV-cache enables efficient autoregressive generation. Scales predictably with compute.

Models: GPT-2/3/4, LLaMA, Mistral, Claude, Gemini.

Encoder-Decoder (T5-style)

Architecture:

Encoder: $L_e$ layers with bidirectional attention.
Decoder: $L_d$ layers with causal self-attention + cross-attention to encoder.

Training objective: Span Corruption. Replace random spans with sentinel tokens; predict the original spans:

\text{Input: } \text{"The } \langle X \rangle \text{ sat on the } \langle Y \rangle\text{"}

\text{Target: } \langle X \rangle \text{ cat } \langle Y \rangle \text{ mat"}

Strengths: Natural for sequence-to-sequence tasks (translation, summarization); encoder provides rich context for generation.

Limitations: More complex; harder to scale; encoder and decoder must both fit in memory.

Models: T5, BART, mBART, Flan-T5, UL2.

Prefix LM (UniLM-style)

Hybrid: The first $P$ tokens are bidirectional (prefix), remaining tokens are causal:

M_{ij} = \begin{cases}0 & j \leq P \text{ (prefix region)} \\ 0 & i \geq j > P \text{ (causal region)} \\ -\infty & i < j, j > P\end{cases} \tag{5}

Advantage: Single model handles both understanding (prefix) and generation (causal suffix).

Models: UniLM, PaLM (partially), U-PaLM.

Mixture of Experts (MoE)

Replace the dense FFN with a sparse set of $E$ expert FFNs. A router selects top- $k$ experts per token:

\text{MoE}(\mathbf{x}) = \sum_{i=1}^k g_i(\mathbf{x}) \cdot \text{FFN}_{e_i}(\mathbf{x}), \tag{6}

where $g_i$ are the router weights and $e_i$ are the selected expert indices.

Properties:

Total parameters: $E \times$ larger than dense model.
Active parameters per token: same as dense (only $k$ experts fire).
Training compute scales sublinearly with parameter count.

Router: $\text{softmax}(\mathbf{W}_r\mathbf{x})$ → select top- $k$ entries.

Load balancing loss: Encourages uniform expert utilization to prevent "expert collapse":

\mathcal{L}_{\text{balance}} = E \cdot \sum_{i=1}^E f_i \cdot P_i, \tag{7}

where $f_i$ = fraction of tokens routed to expert $i$ , $P_i$ = average router probability for expert $i$ .

Models: Mixtral (8x7B), Switch Transformer, GShard, DeepSeek-MoE.

State Space Models (Mamba)

Replace attention entirely with a recurrent state-space computation:

\mathbf{h}_t = \mathbf{A}\mathbf{h}_{t-1} + \mathbf{B}\mathbf{x}_t, \quad y_t = \mathbf{C}\mathbf{h}_t, \tag{8}

where $\mathbf{A}, \mathbf{B}, \mathbf{C}$ are learned (and input-dependent in Mamba's selective SSM).

Complexity: $O(T)$ for sequential processing; $O(T\log T)$ via parallel scan during training.

No KV-cache needed: The fixed-size state $\mathbf{h}$ replaces the growing KV-cache.

Tradeoff: Competitive with transformers for language modeling; less proven for tasks requiring precise retrieval of past tokens (in-context learning).

Models: Mamba, Mamba-2, Jamba (hybrid Mamba + attention).

Architecture Comparison

Encoder-only: Bidirectional, MLM training, best for understanding tasks, cannot generate
Decoder-only: Causal, CLM training, best for generation, scales best, dominant for LLMs
Encoder-decoder: Bidirectional encoder + causal decoder, span corruption, best for seq2seq
MoE: Sparse computation, scales parameters cheaply, same active compute
SSM: Linear complexity, no attention matrix, fixed-size state, emerging alternative

Common Pitfalls

Pitfall 1. Using encoder-only models for generation. BERT cannot generate text naturally; you'd need iterative masking/infilling which is slow and awkward.

Pitfall 2. Assuming MoE = free parameters. While active compute is constant, memory for all expert weights must fit on GPUs. Communication costs for distributed experts are significant.

Pitfall 3. Declaring SSMs will replace transformers. Current evidence shows hybrid architectures (Jamba) work best; pure SSMs struggle with precise recall tasks.

Summary

Decoder-only (causal) dominates LLMs due to simplicity, scalability, and emergent abilities.
Encoder-only excels at understanding tasks with bidirectional context.
Encoder-decoder is natural for sequence-to-sequence but harder to scale.
MoE provides more parameters at constant active compute — key scaling strategy.
SSMs offer linear complexity but trade off precise retrieval ability.

Exercises

Exercise 1. Write the attention mask matrix for a prefix LM with prefix length $P=3$ and total length $T=5$ .

Exercise 2. For a MoE with $E=8$ experts and top- $k=2$ routing, compute the ratio of total to active parameters.

Exercise 3. Derive the parallel scan algorithm for computing the SSM recurrence (equation 8) in $O(T\log T)$ .

Exercise 4. Prove that a decoder-only model with a bidirectional prefix is strictly more expressive than a pure causal model (for the prefix portion).

Exercise 5. Compute the FLOPs per token for a dense model with $d_{ff}=4d$ vs a MoE with $E=8, k=2, d_{ff}=4d$ per expert.