We’ve seen that orthogonal and orthonormal bases make computations much simpler\u2014projections, coordinates, and decompositions all have clean formulas. But how do we obtain such bases in practice?
The Problem: Given an arbitrary basis \(\{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n\}\) for a subspace (or vector space), find an orthonormal basis for the same space.
The Solution: The Gram-Schmidt Process is an algorithm that systematically converts any basis into an orthonormal basis. It works by processing vectors one at a time, removing components that are not orthogonal to the vectors already processed.
QR Factorization: The Gram-Schmidt process underlies the QR factorization \(A = QR\text{,}\) where \(Q\) has orthonormal columns and \(R\) is upper triangular. This is essential for solving least squares problems numerically.
Starting with \(\mathbf{v}_1\text{,}\) we keep it (after normalizing). For \(\mathbf{v}_2\text{,}\) we subtract its projection onto \(\mathbf{v}_1\) to make it orthogonal. For \(\mathbf{v}_3\text{,}\) we subtract its projections onto both \(\mathbf{v}_1\) and \(\mathbf{v}_2\text{.}\) Continuing this way, we build up an orthogonal set that spans the same spaces as the original vectors.
Interpretation: Each \(\mathbf{w}_k\) is obtained by taking \(\mathbf{v}_k\) and subtracting off its components in the directions of all previous \(\mathbf{w}_j\)’s. This leaves only the part of \(\mathbf{v}_k\) that is orthogonal to the span of \(\{\mathbf{w}_1, \ldots, \mathbf{w}_{k-1}\}\text{.}\)
Let \(\mathbf{u}_{i}=\frac{\mathbf{w}_{i}}{\left\|\mathbf{w}_{i}\right\|}\) for each \(i\text{.}\) Then \(B^{\prime \prime}=\left\{\mathbf{u}_{1}, \mathbf{u}_{2}, \ldots, \mathbf{u}_{m}\right\}\) is an orthonormal basis for \(W\text{.}\)
The spans are preserved at each step: \(\operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\} = \operatorname{span}\{\mathbf{u}_1, \ldots, \mathbf{u}_k\}\text{.}\)
The Gram-Schmidt process has a beautiful geometric interpretation: at each step, we project out the "old" components and keep only the "new" orthogonal component. Let’s visualize this step-by-step in \(\mathbb{R}^2\text{.}\)
Normalize: Convert to unit vectors: \(\mathbf{u}_1 = \frac{\mathbf{w}_1}{\|\mathbf{w}_1\|}\text{,}\)\(\mathbf{u}_2 = \frac{\mathbf{w}_2}{\|\mathbf{w}_2\|}\)
Projection removal: At each step, we subtract off the "parallel" components (projections) to isolate the "perpendicular" component that adds a new orthogonal direction.
Span preservation: At every step \(k\text{,}\) we have \(\operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\} = \operatorname{span}\{\mathbf{w}_1, \ldots, \mathbf{w}_k\} = \operatorname{span}\{\mathbf{u}_1, \ldots, \mathbf{u}_k\}\text{.}\) The vectors span the same nested sequence of subspaces.
Why it works: If \(\{\mathbf{v}_1, \ldots, \mathbf{v}_n\}\) are linearly independent, each \(\mathbf{v}_k\) has a non-zero component orthogonal to \(\operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_{k-1}\}\text{.}\) That component becomes \(\mathbf{w}_k\text{.}\)
Example5.3.5.Orthonormalizing a Basis in \(\mathbb{R}^4\).
Let \(S=\{(1,2,-1,0),(2,2,0,1),(1,1,-1,0)\}\) and \(W=\operatorname{span}(S)\text{.}\) Find an orthonormal basis of \(W\) using the Gram-Schmidt process.
