Definition 5.3.1. Inner Product Space.
A vector space \(V\) with an inner product is called an inner product space.
Section 5.3 Inner product space
Subsection 5.3.1 Abstract Vector Space
-
Let \(\mathbf{P}_n(x)=\{a_0+a_1x+\ldots+a_nx^n|a_i\in \mathbb{R}\}\) is a vector space with two operations\begin{align*} (a_0+a_1x \amp +\ldots+a_nx^n) +(b_0+b_1x+\ldots+b_nx^n) \\ \amp = (a_0+b_0)+(a_1+b_1)x+\ldots+(a_n+b_n)x^n \end{align*}\begin{equation*} k(a_0+a_1x+\ldots+a_nx^n)=(k a_0)+(k a_1)x+\ldots+(k a_n)x^n \end{equation*}
-
Define \(C[a, b]\) be the set of all real-valued continuous functions defined on the interval \([a,b]\text{.}\) \(C[a,b]\) is a vector space with operations\begin{equation*} (f+g)(x)=f(x)+g(x) \text{ and } (c f)(x)=c[f(x)]\text{.} \end{equation*}
Subsection 5.3.2 Inner Product Space
Let \(\mathbf{u}, \mathbf{v}\text{,}\) and \(\mathbf{w}\) be vectors in a vector space \(V\text{,}\) and let \(c\) be any scalar. An inner product on \(V\) is a function that associates a real number \(\langle\mathbf{u}, \mathbf{v}\rangle\) with each pair of vectors \(\mathbf{u}\) and \(\mathbf{v}\) and satisfies the axioms listed below.
-
\(\displaystyle \langle\mathbf{u}, \mathbf{v}\rangle=\langle\mathbf{v}, \mathbf{u}\rangle\)
-
\(\displaystyle \langle\mathbf{u}, \mathbf{v}+\mathbf{w}\rangle=\langle\mathbf{u}, \mathbf{v}\rangle+\langle\mathbf{u}, \mathbf{w}\rangle\)
-
\(\displaystyle c\langle\mathbf{u}, \mathbf{v}\rangle=\langle c \mathbf{u}, \mathbf{v}\rangle\)
-
\(\langle\mathbf{v}, \mathbf{v}\rangle \geq 0\text{,}\) and \(\langle\mathbf{v}, \mathbf{v}\rangle=0\) if and only if \(\mathbf{v}=\mathbf{0}\text{.}\)
Lemma 5.3.2.
-
In \(\mathbf{P}_n(x)\text{,}\) the function\begin{equation*} \langle p_1(x), p_2(x) \rangle=a_0b_0+a_1b_1+\ldots+a_nb_n \end{equation*}is an inner product of \(\mathbf{P}_n(x)\text{,}\) where \(p_1(x)=a_0+a_1x+\ldots+a_nx^{n}, p_2(x)=b_0+b_1x+\ldots+b_nx^{n}.\)
-
In \(C[a,b]\text{,}\) the function\begin{equation*} \langle f(x),g(x)\rangle=\int\limits_{a}^{b} f(x)g(x)dx \end{equation*}is an inner product of \(C[a,b]\text{.}\)
Subsection 5.3.3 Gram-Schmidt Process in an Inner Product Space
Proposition 5.3.3.
Theorem 5.3.4.
Let \(S=\{\mathbf{v}_1,\mathbf{v}_2,\ldots, \mathbf{v}_n\}\) be a basis of an inner product space \(V\text{.}\)
-
Let \(B^{\prime}=\left\{\mathbf{w}_{1}, \mathbf{w}_{2}, \ldots, \mathbf{w}_{n}\right\}\text{,}\) where\begin{align*} w_1\amp=\mathbf{v}_{1} \\ w_2\amp=\mathbf{v}_{2}-\frac{\langle\mathbf{v}_{2}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}\\ w_3\amp=\mathbf{v}_{3}-\frac{\langle\mathbf{v}_{3}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}-\frac{\langle\mathbf{v}_{3}, \mathbf{w}_{2}\rangle}{\langle \mathbf{w}_{2}, \mathbf{w}_{2}\rangle} \mathbf{w}_{2} \\ \amp\vdots\\ w_m\amp=\mathbf{v}_{n}-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{2}\rangle}{\langle \mathbf{w}_{2}, \mathbf{w}_{2}\rangle} \mathbf{w}_{2}-\ldots-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{n-1}\rangle}{\langle \mathbf{w}_{n-1}, \mathbf{w}_{n-1}\rangle} \mathbf{w}_{n-1} \end{align*}\(B'\) is an orthogonal basis of \(W\text{.}\)
-
Let \(\mathbf{u}_{i}=\frac{\mathbf{w}_{i}}{\left\|\mathbf{w}_{i}\right\|}\text{.}\) Then \(B^{\prime \prime}=\left\{\mathbf{u}_{1}, \mathbf{u}_{2}, \ldots, \mathbf{u}_{n}\right\}\) is an orthonormal basis for \(W\text{.}\) Also, for \(k=1,2, \ldots, m\text{,}\)\begin{equation*} \operatorname{span}\left\{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots, \mathbf{v}_{k}\right\}=\operatorname{span}\left\{\mathbf{u}_{1}, \mathbf{u}_{2}, \ldots, \mathbf{u}_{k}\right\} \end{equation*}
Example: In \(C[0,2]\text{,}\) find an orthonormal basis of the space \(W=\operatorname{span}\{1,x,x^{2}\}\text{.}\)
Subsection 5.3.4 QR Decomposition
The Gram-Schmidt process has a matrix interpretation called QR decomposition. For any \(m \times n\) matrix \(A\) with linearly independent columns (full column rank), we can factor \(A\) as the product of an orthogonal matrix and an upper triangular matrix.
Definition 5.3.5. QR Decomposition.
Let \(A\) be an \(m \times n\) matrix with linearly independent columns. Then \(A\) can be factored as
\begin{equation*}
A = QR
\end{equation*}
where \(Q\) is an \(m \times n\) matrix with orthonormal columns, and \(R\) is an \(n \times n\) upper triangular matrix with positive diagonal entries.
Connection to Gram-Schmidt: If we apply the Gram-Schmidt process to the columns of \(A = [\mathbf{a}_1 | \mathbf{a}_2 | \cdots | \mathbf{a}_n]\text{,}\) we obtain orthonormal vectors \(\mathbf{q}_1, \mathbf{q}_2, \ldots, \mathbf{q}_n\text{.}\) These form the columns of \(Q = [\mathbf{q}_1 | \mathbf{q}_2 | \cdots | \mathbf{q}_n]\text{.}\)
The matrix \(R\) encodes the coefficients from the Gram-Schmidt process. Specifically, when we write
\begin{align*}
\mathbf{a}_1 &= r_{11}\mathbf{q}_1\\
\mathbf{a}_2 &= r_{12}\mathbf{q}_1 + r_{22}\mathbf{q}_2\\
\mathbf{a}_3 &= r_{13}\mathbf{q}_1 + r_{23}\mathbf{q}_2 + r_{33}\mathbf{q}_3\\
&\vdots
\end{align*}
these coefficients \(r_{ij}\) become the entries of the upper triangular matrix \(R\text{.}\)
Theorem 5.3.6. Existence and Uniqueness of QR Decomposition.
Every \(m \times n\) matrix \(A\) with linearly independent columns has a QR decomposition \(A = QR\) where \(Q\) has is an
orthogonal matrix and \(R\) is upper triangular matrix.
Example: Compute the QR decomposition of a \(3 \times 3\) matrix
\begin{equation*}
A = \begin{pmatrix}
1 \amp 1 \amp 0\\
1 \amp 0 \amp 1\\
0 \amp 1 \amp 1
\end{pmatrix}\text{.}
\end{equation*}
Geometric Interpretation: The QR decomposition gives us two complementary views of the column space:
-
\(Q\) provides an orthonormal basis for \(\text{Col}(A)\)
-
\(R\) describes how to express the original columns in terms of this orthonormal basis
-
The transformation from \(A\) to \(Q\) preserves all geometric properties (lengths, angles, etc.) within the column space
Connection to Previous Subsection: QR decomposition is precisely the matrix formulation of the Gram-Schmidt process. While Gram-Schmidt gives us the step-by-step procedure for orthogonalization, QR decomposition packages this into a compact matrix factorization. Both approaches:
-
Start with linearly independent vectors
-
Produce an orthonormal basis for the same subspace
-
Preserve the spanning relationships between vectors
-
Enable efficient computation of projections and least squares solutions
