Vectors, Spans & Linear Independence

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Vector Spaces: Definition and Axioms
6. Subspaces and the Zero Vector
7. Linear Combinations
8. Span of a Set of Vectors
9. Linear Independence and Dependence
10. The Exchange Lemma and Basis Uniqueness
11. Basis and Dimension
12. Inner Products, Norms, and Similarity
13. Cosine Similarity and the Geometry of Attention
14. The Matrix Formulation of Independence
15. Worked Examples
16. Connection to Machine Learning
17. Common Pitfalls and Misconceptions
18. Research Perspective
19. Summary of Takeaways
20. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. State and verify the vector space axioms for concrete examples arising in machine learning ($\mathbb{R}^n$, matrix spaces, function spaces).
2. Prove that the span of any set of vectors is a subspace, and characterize linear dependence as redundancy among directions.
3. Apply the linear independence criterion $\sum_i \alpha_i \mathbf{v}_i = \mathbf{0} \implies \alpha_i = 0$ to determine whether a set of feature vectors or embeddings is redundant.
4. Define a basis, prove that coordinates with respect to a basis are unique, and explain why dimension is well-defined.
5. Derive the Cauchy–Schwarz inequality and use it to define cosine similarity and the angle between representations.
6. Explain how dot-product attention computes similarity-weighted linear combinations of value vectors.

Prerequisites

This is the opening chapter of Volume I. No prior chapters are required. Familiarity with basic set notation, real numbers, and elementary algebra is assumed.

Notation

• $\mathbb{F}$ — Field (typically $\mathbb{R}$ or $\mathbb{C}$)
• $V, W$ — Vector spaces; $\mathbf{0}$ the zero vector
• $\mathbf{v}_i, \boldsymbol{\alpha}$ — Vectors and coefficient vector
• $\text{span}(S)$ — Set of all linear combinations of $S$
• $\dim(V)$ — Dimension of $V$
• $\langle \mathbf{u}, \mathbf{v} \rangle$ — Inner product; $\ — \mathbf{v}\ —$ induced norm
• $\mathbf{V}, \mathbf{V}^T\mathbf{V}$ — Matrix of column vectors and Gram matrix

Core Intuition

Machine learning operates almost entirely inside vector spaces. A dataset of $n$ examples with $d$ features is an $n \times d$ matrix whose rows and columns are vectors. A neural network's parameters form a long vector in $\mathbb{R}^p$. An embedding maps discrete tokens into a continuous vector space where geometric proximity encodes semantic similarity.

Before we can discuss matrices, optimization, or attention, we need a precise language for three questions:

• Which outputs can we reach? This is the span — all vectors obtainable as linear combinations of given directions.
• Are any directions redundant? This is linear independence — no vector in the set can be built from the others.
• What is the minimal complete description? This is a basis — an independent spanning set whose size defines dimension.

These concepts are not merely algebraic formalism. They determine whether a linear regression problem has a unique solution, whether an embedding space has redundant dimensions, and whether an attention head can represent distinct contextual relationships. The rest of this chapter develops these ideas with full proofs.

Series context. This is Part I of Chapter 1 (Linear Algebra) in Volume I: Foundations. Subsequent parts in this chapter cover Matrix Operations & Linear Transformations, Eigenvalues & Eigenvectors, and the Singular Value Decomposition.

Vector Spaces: Definition and Axioms

Definition 1 (Field). A field $\mathbb{F}$ is a set equipped with addition and multiplication satisfying the usual axioms. In this curriculum we work primarily with $\mathbb{F} = \mathbb{R}$ (real numbers) and occasionally $\mathbb{F} = \mathbb{C}$ (complex numbers).

Definition 2 (Vector Space). A vector space over $\mathbb{F}$ is a set $V$ together with two operations:

• Vector addition: $+: V \times V \to V$, written $\mathbf{u} + \mathbf{v}$
• Scalar multiplication: $\cdot: \mathbb{F} \times V \to V$, written $\alpha \mathbf{v}$

such that for all $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and $\alpha, \beta \in \mathbb{F}$, the following axioms hold.

Additive structure:

Scalar multiplication structure:

Definition 3 (Vector). An element $\mathbf{v} \in V$ is called a vector. When $V = \mathbb{R}^n$, we write $\mathbf{v} = (v_1, \ldots, v_n)^T$ and call $v_i$ the components of $\mathbf{v}$.

Example 1 ($\mathbb{R}^n$). The set $\mathbb{R}^n = \{(x_1, \ldots, x_n) : x_i \in \mathbb{R}\}$ with component-wise addition and scalar multiplication is a vector space. Every feature vector in supervised learning lives in $\mathbb{R}^d$ for some feature dimension $d$.

Example 2 (Matrix space). The set $\mathbb{R}^{m \times n}$ of $m \times n$ real matrices, with matrix addition and scalar multiplication, is a vector space. A neural network's weight matrices collectively span a subspace of a much larger matrix space.

Example 3 (Function space). The set $C([0,1], \mathbb{R})$ of continuous real-valued functions on $[0,1]$, with pointwise addition and scalar multiplication, is an infinite-dimensional vector space. Reproducing kernel Hilbert spaces used in kernel methods are subspaces of such function spaces.

Proposition 1. In any vector space $V$, the zero vector $\mathbf{0}$ is unique, and for each $\mathbf{v} \in V$, the additive inverse $-\mathbf{v}$ is unique.

Proof. Suppose $\mathbf{0}$ and $\mathbf{0}'$ are both additive identities. Then $\mathbf{0} = \mathbf{0} + \mathbf{0}' = \mathbf{0}'$. For inverses, if $\mathbf{v} + \mathbf{w}_1 = \mathbf{0}$ and $\mathbf{v} + \mathbf{w}_2 = \mathbf{0}$, then $\mathbf{w}_1 = \mathbf{w}_1 + \mathbf{0} = \mathbf{w}_1 + (\mathbf{v} + \mathbf{w}_2) = (\mathbf{w}_1 + \mathbf{v}) + \mathbf{w}_2 = \mathbf{0} + \mathbf{w}_2 = \mathbf{w}_2$. $\blacksquare$

Subspaces and the Zero Vector

Definition 4 (Subspace). A subset $W \subseteq V$ is a subspace of $V$ if $W$ is itself a vector space under the same operations — equivalently, if:

1. $\mathbf{0} \in W$
2. $\mathbf{u}, \mathbf{v} \in W \implies \mathbf{u} + \mathbf{v} \in W$ (closed under addition)
3. $\mathbf{u} \in W, \alpha \in \mathbb{F} \implies \alpha \mathbf{u} \in W$ (closed under scalar multiplication)

Proposition 2. A nonempty subset $W \subseteq V$ is a subspace if and only if it is closed under linear combinations: $\mathbf{u}, \mathbf{v} \in W, \alpha, \beta \in \mathbb{F} \implies \alpha\mathbf{u} + \beta\mathbf{v} \in W$.

This characterization is the workhorse for proving that various constructions (span, null space, column space) yield subspaces.

Linear Combinations

Definition 5 (Linear Combination). Given vectors $\mathbf{v}_1, \ldots, \mathbf{v}_k \in V$ and scalars $\alpha_1, \ldots, \alpha_k \in \mathbb{F}$, the vector

is called a linear combination of $\mathbf{v}_1, \ldots, \mathbf{v}_k$.

Important equation. Equation (3) is the fundamental building block of linear models. A prediction $\hat{y} = \mathbf{w}^T \mathbf{x} = \sum_{j=1}^d w_j x_j$ is a scalar obtained from a linear combination of features followed by an inner product. Each layer of a neural network computes linear combinations of activations before applying a nonlinearity.

Proposition 3. The set of all linear combinations of $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ is the smallest subspace of $V$ containing $\mathbf{v}_1, \ldots, \mathbf{v}_k$.

We will denote this set by $\text{span}(\mathbf{v}_1, \ldots, \mathbf{v}_k)$ and prove in the next section that it is indeed a subspace.

Span of a Set of Vectors

Definition 6 (Span). The span of $S = \{\mathbf{v}_1, \ldots, \mathbf{v}_k\} \subseteq V$ is

If $S = \emptyset$, we define $\text{span}(\emptyset) = \{\mathbf{0}\}$.

Theorem 1. For any set $S \subseteq V$, $\text{span}(S)$ is a subspace of $V$.

Proof. We verify the three subspace conditions.

(i) Contains zero: For $S = \{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$, set all $\alpha_i = 0$. Then $\sum_i 0 \cdot \mathbf{v}_i = \mathbf{0} \in \text{span}(S)$.

(ii) Closed under addition: Let $\mathbf{u} = \sum_i \alpha_i \mathbf{v}_i$ and $\mathbf{w} = \sum_i \beta_i \mathbf{v}_i$ belong to $\text{span}(S)$. Then

(iii) Closed under scalar multiplication: Let $\mathbf{u} = \sum_i \alpha_i \mathbf{v}_i \in \text{span}(S)$ and $c \in \mathbb{F}$. Then

Geometric interpretation in $\mathbb{R}^3$. For nonzero $\mathbf{v}_1$:

• $\text{span}(\mathbf{v}_1)$ is a line through the origin.
• $\text{span}(\mathbf{v}_1, \mathbf{v}_2)$ is a plane through the origin, provided $\mathbf{v}_2 \notin \text{span}(\mathbf{v}_1)$.
• $\text{span}(\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3) = \mathbb{R}^3$ if the three vectors are not coplanar.

Visual description. Imagine three flashlight beams emanating from the origin. Each beam is $\text{span}(\mathbf{v}_i)$. The span of two non-parallel beams fills a flat sheet (a plane). The span of three non-coplanar beams fills all of space. Machine learning representations similarly "fill" a subspace of $\mathbb{R}^d$; the dimension of that subspace measures how many independent degrees of freedom the representation possesses.

Linear Independence and Dependence

Definition 7 (Linear Independence). Vectors $\mathbf{v}_1, \ldots, \mathbf{v}_k \in V$ are linearly independent if the only solution to

is the trivial solution $\alpha_1 = \alpha_2 = \cdots = \alpha_k = 0$.

Definition 8 (Linear Dependence). The set is linearly dependent if there exist scalars $\alpha_1, \ldots, \alpha_k$, not all zero, satisfying equation (7).

Important equation. Equation (7) is the master test for independence. In machine learning, if columns of a feature matrix satisfy a nontrivial version of (7), the features carry redundant information and the model may be unidentifiable.

Theorem 2. The set $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ is linearly dependent if and only if at least one vector is a linear combination of the others.

Proof. ($\Rightarrow$) Suppose $\sum_i \alpha_i \mathbf{v}_i = \mathbf{0}$ with not all $\alpha_i = 0$. Let $j$ be an index with $\alpha_j \neq 0$. Then

($\Leftarrow$) Suppose $\mathbf{v}_j = \sum_{i \neq j} \beta_i \mathbf{v}_i$. Rearranging,

which is a nontrivial linear combination equal to zero (the coefficient of $\mathbf{v}_j$ is $-1 \neq 0$). $\blacksquare$

Theorem 3 (Steinitz Exchange Lemma — Preview). If $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ is linearly independent and each $\mathbf{v}_i \in \text{span}(\mathbf{w}_1, \ldots, \mathbf{w}_m)$, then $k \leq m$.

This fundamental result implies that any linearly independent set in a space spanned by $m$ vectors has at most $m$ elements. We will use it to prove that all bases have the same size.

Corollary 1. In $\mathbb{R}^n$, any set of more than $n$ vectors is linearly dependent.

Proof. Any $k$ vectors in $\mathbb{R}^n$ belong to $\text{span}(\mathbf{e}_1, \ldots, \mathbf{e}_n)$ where $\{\mathbf{e}_i\}$ is the standard basis. If $k > n$, the exchange lemma applied to an independent subset (if one existed with $k > n$ elements) yields a contradiction. $\blacksquare$

The Exchange Lemma and Basis Uniqueness

We now prove a version of the exchange lemma, which underpins the concept of dimension.

Lemma 1 (Exchange Lemma). Let $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ be linearly independent and suppose $\mathbf{v}_k \in \text{span}(\mathbf{w}_1, \ldots, \mathbf{w}_m)$. Then there exists an index $j \in \{1, \ldots, m\}$ such that $\{\mathbf{v}_1, \ldots, \mathbf{v}_{k-1}, \mathbf{w}_j\}$ is linearly independent.

Proof. Write $\mathbf{v}_k = \sum_{j=1}^m \beta_j \mathbf{w}_j$ with not all $\beta_j = 0$ (since $\mathbf{v}_k \neq \mathbf{0}$ by independence of the $\mathbf{v}_i$). Choose $j$ with $\beta_j \neq 0$. Suppose $\alpha_1 \mathbf{v}_1 + \cdots + \alpha_{k-1}\mathbf{v}_{k-1} + \gamma \mathbf{w}_j = \mathbf{0}$. Substituting for $\mathbf{w}_j$ from the expression for $\mathbf{v}_k$ yields a linear combination of $\mathbf{v}_1, \ldots, \mathbf{v}_k$ equal to zero. Independence of the $\mathbf{v}_i$ forces $\alpha_i = 0$ and then $\gamma = 0$. $\blacksquare$

Theorem 4. All bases of a finite-dimensional vector space have the same cardinality.

Proof sketch. Let $\mathcal{B}_1 = \{\mathbf{b}_1, \ldots, \mathbf{b}_n\}$ and $\mathcal{B}_2 = \{\mathbf{c}_1, \ldots, \mathbf{c}_m\}$ be two bases. Since $\mathcal{B}_1$ is independent and spans the space, and each $\mathbf{b}_i$ is in $\text{span}(\mathcal{B}_2)$, the exchange lemma gives $n \leq m$. Reversing roles gives $m \leq n$. Hence $n = m$. $\blacksquare$

Basis and Dimension

Definition 9 (Basis). A basis for a vector space $V$ is a set $\mathcal{B} \subseteq V$ that is:

1. Linearly independent, and
2. Spanning: $\text{span}(\mathcal{B}) = V$.

Definition 10 (Dimension). If $V$ has a finite basis, the dimension of $V$, written $\dim(V)$, is the number of vectors in any basis. If no finite basis exists, $V$ is infinite-dimensional.

Theorem 5 (Unique Representation). If $\mathcal{B} = \{\mathbf{b}_1, \ldots, \mathbf{b}_n\}$ is a basis for $V$, then every $\mathbf{v} \in V$ has a unique representation

Proof. Existence follows from spanning. For uniqueness, suppose $\mathbf{v} = \sum_i c_i \mathbf{b}_i = \sum_i d_i \mathbf{b}_i$. Then $\sum_i (c_i - d_i)\mathbf{b}_i = \mathbf{0}$, and independence gives $c_i = d_i$ for all $i$. $\blacksquare$

The scalars $(c_1, \ldots, c_n)$ are the coordinates of $\mathbf{v}$ with respect to $\mathcal{B}$.

Standard basis. The standard basis of $\mathbb{R}^n$ is $\{\mathbf{e}_1, \ldots, \mathbf{e}_n\}$ where $\mathbf{e}_i$ has a 1 in position $i$ and zeros elsewhere. Every $\mathbf{x} \in \mathbb{R}^n$ has coordinates $(x_1, \ldots, x_n)$ in this basis.

Learned bases in machine learning. Methods such as PCA, the SVD, and autoencoders learn alternative bases (or overcomplete frames) adapted to data structure. See Eigenvalues & Eigenvectors for the spectral construction of principal components.

Inner Products, Norms, and Similarity

For geometric reasoning in $\mathbb{R}^n$, we equip the vector space with an inner product.

Definition 11 (Inner Product). An inner product on $\mathbb{R}^n$ is a function $\langle \cdot, \cdot \rangle : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$ satisfying, for all $\mathbf{u}, \mathbf{v}, \mathbf{w} \in \mathbb{R}^n$ and $\alpha \in \mathbb{R}$:

1. Symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$
2. Linearity in the first argument: $\langle \alpha \mathbf{u} + \mathbf{v}, \mathbf{w} \rangle = \alpha \langle \mathbf{u}, \mathbf{w} \rangle + \langle \mathbf{v}, \mathbf{w} \rangle$
3. Positive definiteness: $\langle \mathbf{v}, \mathbf{v} \rangle \geq 0$, with equality iff $\mathbf{v} = \mathbf{0}$

The Euclidean inner product (dot product) is $\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} = \sum_{i=1}^n u_i v_i$.

Definition 12 (Norm). The norm induced by an inner product is

Theorem 6 (Cauchy–Schwarz Inequality). For all $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$,

Equality holds if and only if $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent.

Proof. If $\mathbf{v} = \mathbf{0}$, the inequality is trivial. Otherwise, for any $t \in \mathbb{R}$, positive definiteness gives

This quadratic in $t$ has nonpositive discriminant:

which is precisely (12). Equality occurs when $\mathbf{u} + t\mathbf{v} = \mathbf{0}$ for some $t$, i.e., when $\mathbf{u}$ and $\mathbf{v}$ are dependent. $\blacksquare$

Definition 13 (Angle and Cosine Similarity). For nonzero $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$, the cosine similarity is

where $\theta \in [0, \pi]$ is the angle between $\mathbf{u}$ and $\mathbf{v}$. By (12), $\text{sim}(\mathbf{u}, \mathbf{v}) \in [-1, 1]$.

Important equation. Equation (15) measures similarity by direction alone, ignoring magnitude. Two embedding vectors pointing in nearly the same direction are semantically similar even if one has larger norm (e.g., due to frequency effects in language models).

Proposition 4 (Parallelogram Law). For all $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$,

This identity reflects the geometry of vector addition and is useful in analyzing distances in embedding spaces.

Cosine Similarity and the Geometry of Attention

Self-attention, developed in the Transformer architecture (Vaswani et al., 2017), computes similarity-weighted linear combinations of value vectors. The query–key dot product is cosine similarity when vectors are normalized.

Setup. Given query, key, and value vectors $\mathbf{q}, \mathbf{k}, \mathbf{v} \in \mathbb{R}^d$ for a single attention head, the scaled dot-product attention score is

For unit-norm queries and keys ($\|\mathbf{q}\| = \|\mathbf{k}\| = 1$), $s = \cos\theta / \sqrt{d}$ where $\theta$ is the angle between $\mathbf{q}$ and $\mathbf{k}$.

Attention weights. For a sequence of $T$ tokens with queries $\mathbf{q}_i$, keys $\mathbf{k}_j$, and values $\mathbf{v}_j$, the attention output for token $i$ is

Interpretation. Equation (18) is a linear combination (Definition 5) of value vectors, where the coefficients $\alpha_{ij}$ are positive and sum to 1 (a convex combination). The weights depend on the similarity between query $\mathbf{q}_i$ and key $\mathbf{k}_j$ via equation (17). Thus attention selects which value directions to combine based on geometric alignment in representation space.

Visual description. Imagine token $i$ casting a query vector into a shared space. Each other token $j$ presents a key vector. The dot product measures how well the query "aligns" with each key — like projecting a search direction onto catalog entries. Softmax converts these alignment scores into a probability distribution over tokens. The output is then a weighted blend of value vectors, pulling information from tokens whose keys match the query.

The scaling factor $\sqrt{d}$ prevents dot products from growing with dimension, which would drive softmax into extreme (nearly one-hot) distributions. We develop the full Transformer theory in Volume III, Part IV.

The Matrix Formulation of Independence

Given $\mathbf{v}_1, \ldots, \mathbf{v}_k \in \mathbb{R}^n$, form the matrix

Then $\sum_i \alpha_i \mathbf{v}_i = \mathbf{V}\boldsymbol{\alpha}$ where $\boldsymbol{\alpha} = (\alpha_1, \ldots, \alpha_k)^T$. The vectors are linearly independent iff

i.e., $\text{null}(\mathbf{V}) = \{\mathbf{0}\}$.

Proposition 5. For $\mathbf{V} \in \mathbb{R}^{n \times k}$ with $k \leq n$, the following are equivalent:

1. Columns of $\mathbf{V}$ are linearly independent.
2. $\text{rank}(\mathbf{V}) = k$.
3. $\mathbf{V}^T \mathbf{V} \in \mathbb{R}^{k \times k}$ is invertible.

Proof. (1 $\iff$ 2) is the definition of rank as column rank. (2 $\iff$ 3): $\mathbf{V}^T \mathbf{V}$ is $k \times k$ and $\text{rank}(\mathbf{V}^T \mathbf{V}) = \text{rank}(\mathbf{V})$. A square matrix is invertible iff its rank equals its size. $\blacksquare$

The Gram matrix $\mathbf{V}^T \mathbf{V}$ appears in normal equations, kernel methods, and the analysis of feature redundancy. When it is singular, the columns of $\mathbf{V}$ are dependent.

Worked Examples

Example 4: Testing Independence in $\mathbb{R}^3$

Determine whether the following vectors are linearly independent:

Solution. We solve $\mathbf{V}\boldsymbol{\alpha} = \mathbf{0}$ with $\mathbf{V} = [\mathbf{v}_1 | \mathbf{v}_2 | \mathbf{v}_3]$:

Row reduction yields rank 2, so there is a free variable. One nontrivial solution is $\boldsymbol{\alpha} = (1, -2, 1)^T$, giving $\mathbf{v}_1 - 2\mathbf{v}_2 + \mathbf{v}_3 = \mathbf{0}$. The vectors are linearly dependent and span a 2-dimensional plane in $\mathbb{R}^3$.

Observation. $\mathbf{v}_3 - \mathbf{v}_2 = \mathbf{v}_2 - \mathbf{v}_1$, so the three vectors lie on a single line of arithmetic progressions — a classic rank-deficient configuration.

Example 5: Cosine Similarity

Let $\mathbf{u} = (1, 0, 0)^T$ and $\mathbf{v} = (1, 1, 0)^T$. Then

The angle is $\theta = 45°$. Normalizing both vectors before computing dot products would yield the same similarity score — the reason many embedding pipelines use $\ell_2$-normalized vectors for retrieval.

Connection to Machine Learning

Feature Redundancy and Identifiability

If the columns of a feature matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ are linearly dependent, then $\mathbf{X}^T \mathbf{X}$ is singular and the normal equations for linear regression lack a unique solution. Multiple parameter vectors $\mathbf{w}$ produce identical predictions $\mathbf{X}\mathbf{w}$. Regularization (Ridge regression adds $\lambda \mathbf{I}$ to $\mathbf{X}^T \mathbf{X}$) restores invertibility by perturbing the Gram matrix — a theme developed in Positive Definite Matrices.

Embedding Dimension and Rank

Word embeddings map a vocabulary of size $|V|$ into $\mathbb{R}^d$. The embedding matrix $\mathbf{E} \in \mathbb{R}^{|V| \times d}$ has rank at most $d$, so all word vectors lie in a $d$-dimensional subspace of $\mathbb{R}^{|V|}$ (when viewed as rows) or $\mathbb{R}^d$ (as columns). Training learns a basis for semantic directions within this subspace.

Attention Head Subspaces

Multi-head attention with $h$ heads and model dimension $d$ partitions representations into $h$ subspaces of dimension $d_k = d/h$. Each head computes attention within its subspace; concatenating heads spans at most $d$ dimensions. The linear-algebraic structure of these subspaces governs representational capacity — see Self-Attention in Volume III.

Common Pitfalls and Misconceptions

Pitfall 1: Confusing linear independence with orthogonality. Independent vectors need not be perpendicular. Orthogonality is a stronger geometric condition requiring $\langle \mathbf{u}, \mathbf{v} \rangle = 0$. Independent vectors can have arbitrary angles.

Pitfall 2: Treating "linearly dependent" as "one vector is a scalar multiple of another." Dependence allows any linear combination, not just scaling. For example, $(1,0,0)$, $(0,1,0)$, and $(1,1,0)$ are dependent because $(1,1,0) = (1,0,0) + (0,1,0)$, yet no vector is a scalar multiple of another.

Pitfall 3: Assuming $\dim(\text{span}(S)) = |S|$. The dimension of the span equals the size of a maximal independent subset, not the size of $S$ itself. Duplicate or redundant vectors do not increase dimension.

Pitfall 4: Ignoring the origin in geometric intuition. Spans and subspaces always contain $\mathbf{0}$. An "offset" subspace such as $\{\mathbf{b} + \text{span}(S)\}$ is an affine set, not a subspace, unless $\mathbf{b} = \mathbf{0}$. Neural network biases introduce exactly such affine shifts.

Pitfall 5: Conflating cosine similarity with Euclidean distance. Two vectors can have high cosine similarity (same direction) but large Euclidean distance (different magnitudes). Retrieval systems must choose the metric appropriate to the task.

Research Perspective

The abstraction of vector spaces dates to Peano (1888) and the modern axiomatic treatment to Banach and Hilbert in functional analysis. The shift from coordinate-based calculations to coordinate-free linear algebra — emphasizing span, independence, and dimension — was essential for twentieth-century physics and statistics.

In machine learning, the vector space viewpoint became central with the development of kernel methods (Schölkopf & Smola, 2002), which embed data implicitly in high-dimensional (often infinite-dimensional) feature spaces. Word2Vec (Mikolov et al., 2013) demonstrated that semantic structure emerges as linear geometry in learned embedding spaces — analogies as vector arithmetic. Transformers (Vaswani et al., 2017) extended this to contextual representations, where attention computes dynamic linear combinations based on query–key similarity.

Modern research continues to probe the effective dimension of learned representations: although models operate in $\mathbb{R}^d$ with $d$ in the thousands, intrinsic dimension estimates often suggest that data manifolds occupy far lower-dimensional subspaces. Understanding span, rank, and independence remains essential for interpreting compression, pruning, and low-rank adaptation methods.

Summary of Takeaways

• Vector space — Set with addition and scalar multiplication satisfying axioms (1)–(2) — Feature spaces, parameter spaces
• Linear combination — $\sum_i \alpha_i \mathbf{v}_i$ — Predictions, layer outputs, attention
• Span — All linear combinations of a set — Representable functions / outputs
• Independence — Only trivial solution to $\sum \alpha_i \mathbf{v}_i = \mathbf{0}$ — Non-redundant features
• Basis — Independent + spanning — Minimal complete representation
• Dimension — Size of any basis — Embedding dim, rank, capacity
• Inner product — Symmetric bilinear form with positive definiteness — Dot products, kernels
• Cosine similarity — $\langle \mathbf{u}, \mathbf{v} \rangle / (\ — \mathbf{u}\ — \ — \mathbf{v}\ — )$ — Attention scores, retrieval

Next article: Matrix Operations & Linear Transformations → — where matrices are defined as linear maps between vector spaces, and composition becomes matrix multiplication.

Exercises

Exercise 1 (Verification). Prove that $\mathbb{R}^{m \times n}$ with matrix addition and scalar multiplication satisfies all vector space axioms (1)–(2).

Exercise 2 (Subspace test). Let $W = \{(\mathbf{x}, \mathbf{y}) \in \mathbb{R}^n \times \mathbb{R}^n : \mathbf{x} = \mathbf{y}\}$ embedded in $\mathbb{R}^{2n}$ via concatenation. Prove that $W$ is a subspace and find a basis. What is $\dim(W)$?

Exercise 3 (Independence). Show that $\{(1,0,0), (1,1,0), (1,1,1)\}$ is a basis for $\mathbb{R}^3$ by verifying independence and that $\dim(\text{span}) = 3$.

Exercise 4 (Cauchy–Schwarz application). For nonzero $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$, prove the triangle inequality $\|\mathbf{u} + \mathbf{v}\| \leq \|\mathbf{u}\| + \|\mathbf{v}\|$ using the Cauchy–Schwarz inequality.

Exercise 5 (Attention algebra). Let $\mathbf{q}, \mathbf{k}_1, \mathbf{k}_2, \mathbf{v}_1, \mathbf{v}_2 \in \mathbb{R}^d$ with $\|\mathbf{q}\| = \|\mathbf{k}_j\| = 1$. Suppose $\langle \mathbf{q}, \mathbf{k}_1 \rangle = 1$ and $\langle \mathbf{q}, \mathbf{k}_2 \rangle = 0$. Compute the attention output (18) when $T = 2$ and softmax is applied over $j \in \{1, 2\}$. Show that the output reduces to $\mathbf{v}_1$ in the limit of large scaling (before softmax saturation, analyze the weights explicitly for finite scaling).

Exercise 6 (Conceptual). Explain why adding a bias term $\mathbf{b}$ to a linear map $\mathbf{x} \mapsto \mathbf{W}\mathbf{x}$ breaks the subspace structure, and describe how augmenting $\mathbf{x}$ with a constant coordinate restores linearity. (Preview of affine transformations in the next article.)

Exercise 7 (Proof). Complete the proof of Theorem 4 (all bases have the same size) using the exchange lemma iteratively.