Skip to main content

Linear Algebra For Undergraduate Students

Yilan(Peter) Tan

Section 5.2 Orthogonal Projection

Why Are Orthogonal Projections So Important?
🔗
Orthogonal projection is one of the most fundamental operations in linear algebra, with applications ranging from computer graphics to machine learning. The key idea is simple but powerful: given a vector and a subspace, find the vector in the subspace that is \"closest\" to the given vector.
🔗
The Closest Point Problem:
🔗
Suppose you have a vector \(\mathbf{b}\) and a subspace \(W\text{.}\) What vector \(\mathbf{w} \in W\) minimizes the distance \(\|\mathbf{b} - \mathbf{w}\|\text{?}\)
🔗
Answer: The orthogonal projection of \(\mathbf{b}\) onto \(W\text{.}\) Geometrically, we \"drop a perpendicular\" from \(\mathbf{b}\) to \(W\text{.}\)
🔗
Why This Matters:
🔗
  • Best Approximation: If we can’t represent \(\mathbf{b}\) exactly in \(W\text{,}\) the projection gives the best approximation.
    🔗
    🔗
  • Least Squares: Solving \(A\mathbf{x} = \mathbf{b}\) when there’s no exact solution means projecting \(\mathbf{b}\) onto the column space of \(A\text{.}\)
    🔗
    🔗
  • Data Compression: Projecting high-dimensional data onto lower-dimensional subspaces (PCA) preserves maximum information.
    🔗
    🔗
  • Computer Graphics: Projecting 3D objects onto 2D screens, computing shadows, and finding closest points on surfaces.
    🔗
    🔗
🔗
The Power of Orthogonal Bases:
🔗
Computing projections is much simpler when we have an orthogonal (or better, orthonormal) basis for the subspace. This motivates the Gram-Schmidt process, which converts any basis into an orthonormal one.
🔗

Subsection 5.2.1 Orthogonal and Orthonormal Sets

Why Study Orthogonal Sets?
🔗
Orthogonal vectors represent independent, perpendicular directions. Working with orthogonal bases simplifies many computations:
  • Projections have simple formulas (no matrix inverses needed!)
    🔗
  • Coordinates are easy to compute (just take inner products)
    🔗
  • Orthogonal matrices preserve lengths and angles
    🔗
  • Many numerical algorithms are more stable with orthogonal bases
    🔗
🔗

Definitions of Orthogonal and Orthonormal.

A set \(S=\{v_1,v_2,\ldots,v_m\}\) of vectors in \(\mathbb{R}^{n}\) is orthogonal when every pair of vectors in \(S\) is orthogonal. That is, \(v_i\cdot v_j=0\) for all \(1\leq i\neq j\leq m\text{.}\)
🔗
If, in addition, each vector in the set is a unit vector, then \(S\) is orthonormal.
🔗
Key Property: Any orthogonal set of nonzero vectors is linearly independent. This is because if \(c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_m\mathbf{v}_m = \mathbf{0}\text{,}\) taking the dot product with \(\mathbf{v}_i\) gives \(c_i\|\mathbf{v}_i\|^2 = 0\text{,}\) so \(c_i = 0\text{.}\)
🔗
🔗

Example 5.2.1.

Show that the set \(S\) below is an orthogonal set of \(\mathbb{R}^{4}\text{.}\)
\begin{equation*} S=\{(2,3,2,-2),(1,0,0,1),(-1,0,2,1),(-1,2,-1,1)\} \end{equation*}
🔗
🔗
One can check \(S\) is orthogonal for every pair of vectors. You may answer this by matrix multiplication.
🔗
Looking Ahead: Since this is an orthogonal set of 4 nonzero vectors in \(\mathbb{R}^4\text{,}\) it is linearly independent and therefore forms a basis. In the next section, we’ll learn the Gram-Schmidt Process, which converts any basis into an orthonormal basis.
🔗
🔗

Subsection 5.2.2 Orthogonal Projection

Let \(\mathbf{u}\) and \(\mathbf{v}\) be vectors in \(\mathbb{R}^{n}\) such that \(\mathbf{v} \neq \mathbf{0}\text{.}\) Then the orthogonal projection of \(\mathbf{u}\) onto \(\mathbf{v}\) is the vector \(k\mathbf{v}\) (\(k\in \mathbb{R}\)) such that \(\|\mathbf{u}-k\mathbf{v}\|\) attains minimum, denoted by \(\operatorname{proj}_{\mathbf{v}} \mathbf{u}\text{.}\)
🔗
A diagram showing vector u decomposed into its projection onto v and the perpendicular component.
Figure 5.2.3. Orthogonal Projection of \(\mathbf{u}\) onto \(\mathbf{v}\)
🔗
Key geometric insight: The projection \(\operatorname{proj}_\mathbf{v}\mathbf{u}\) is the component of \(\mathbf{u}\) in the direction of \(\mathbf{v}\text{.}\) The vector \(\mathbf{u} - \operatorname{proj}_\mathbf{v}\mathbf{u}\) (shown in dashed orange) is perpendicular to \(\mathbf{v}\text{,}\) which is why this is called an "orthogonal" projection. This decomposition \(\mathbf{u} = \operatorname{proj}_\mathbf{v}\mathbf{u} + (\mathbf{u} - \operatorname{proj}_\mathbf{v}\mathbf{u})\) splits \(\mathbf{u}\) into parallel and perpendicular components relative to \(\mathbf{v}\text{.}\)
🔗
Figure 5.2.4. Given a nonzero vector \(\mathbf{w}\) and a vector \(\mathbf{b}\text{,}\) the orthogonal projection of vector \(\mathbf{b}\) on \(w\) is the vector \(\widehat{\mathbf{b}}\text{.}\)
🔗
🔗
Discussion: Describe the relation between the vector \(\mathbf{b}-\widehat{\mathbf{b}}\) and the vector \(\mathbf{w}\text{.}\)
🔗
There is a great observation if the basis is orthonormal.
🔗

Proof.

It follows from Proposition 5.2.5 that
\begin{align*} \hat{w} \amp= (w\cdot v_1)~v_1 + (w\cdot v_2)~v_2 + \cdots + (w\cdot v_m)~v_m\\ \amp = [v_1\, v_2\, \ldots\, v_m]\left[\begin{array}{c} w\cdot v_1\\ w\cdot v_2\\ \ddots \\ w\cdot v_m \end{array}\right]\\ \amp = [v_1\, v_2\, \ldots\, v_m]\left[\begin{array}{c} v_1\cdot w\\ v_2\cdot w\\ \ddots \\ v_m\cdot w \end{array}\right]\\ \amp = [v_1\, v_2\, \ldots\, v_m]\left[\begin{array}{c} v_1^{T}w\\ v_2^{T}w\\ \ddots \\ v_m^{T} w \end{array}\right]\\ \amp = [v_1\, v_2\, \ldots\, v_m]\left[\begin{array}{c} v_1^{T}\\ v_2^{T}\\ \ddots \\ v_m^{T} \end{array}\right]w\\ \amp=\left(QQ^{T}\right)(w) \end{align*}
🔗
🔗
🔗
🔗
PrevTopNext