Inner Product Spaces

Section 5.4 Inner Product Spaces

Why Generalize to Abstract Inner Product Spaces?

So far, we have worked exclusively with the dot product in \(\mathbb{R}^n\text{.}\) While powerful, this limits us to vectors that are sequences of numbers. But many mathematical objects that we want to treat as \"vectors\" are not sequences of numbers\u2014for example:

Polynomials like \(3 + 2x - x^2\)

🔗
Matrices like \(\begin{bmatrix}1 \amp 2\\3 \amp 4\end{bmatrix}\)

🔗
Functions like \(f(x) = \sin(x)\) or \(g(x) = e^x\)

🔗

🔗

The key insight: we can define \"inner products\" on these spaces that satisfy the same algebraic properties as the dot product. Once we do this, all our geometric concepts\u2014length, distance, angle, orthogonality, projection\u2014automatically extend to these abstract spaces!

🔗

The Power of Abstraction:

🔗

By identifying the essential properties of the dot product (commutativity, linearity, positive definiteness) and using them as axioms for a general inner product, we can:

Apply geometric reasoning to spaces of polynomials and functions

🔗
Define orthogonal bases for function spaces (Fourier series!)

🔗
Use least squares to approximate functions (not just fit data)

🔗
Solve differential equations using projection methods

🔗

🔗

This is one of the most beautiful examples of abstraction in mathematics: by understanding the structure that makes Euclidean space \"work,\" we can recreate that structure in entirely different contexts.

🔗

Subsection 5.4.1 Examples of Abstract Vector Spaces

Before defining inner products on abstract spaces, let’s recall some important vector spaces beyond \(\mathbb{R}^n\text{:}\)

🔗

Polynomial Space \(\mathcal{P}_n(x)\text{:}\)
🔗

Let \(\mathcal{P}_n(x)=\{a_0+a_1x+\ldots+a_nx^n\mid a_i\in \mathbb{R}\}\) be the set of all polynomials of degree at most \(n\) with real coefficients. This is a vector space with operations:
🔗

\begin{align*} (a_0+a_1x \amp +\ldots+a_nx^n) +(b_0+b_1x+\ldots+b_nx^n) \\ \amp = (a_0+b_0)+(a_1+b_1)x+\ldots+(a_n+b_n)x^n \end{align*}

🔗

\begin{equation*} k(a_0+a_1x+\ldots+a_nx^n)=(k a_0)+(k a_1)x+\ldots+(k a_n)x^n \end{equation*}

🔗

Why is this useful? Polynomials approximate smooth functions (Taylor series) and are used in numerical analysis, control theory, and computer graphics.
🔗

🔗
Space of Continuous Functions \(C[a, b]\text{:}\)
🔗

Define \(C[a, b]\) to be the set of all real-valued continuous functions defined on the interval \([a,b]\text{.}\) This is a vector space with operations

\begin{equation*} (f+g)(x)=f(x)+g(x) \quad\text{ and }\quad (c f)(x)=c[f(x)]\text{.} \end{equation*}

🔗

Why is this useful? Physical signals (sound, electromagnetic waves) are continuous functions. Understanding their geometry leads to Fourier analysis and signal processing.
🔗

🔗

🔗

These are infinite-dimensional vector spaces\u2014they cannot be described using a finite basis. Yet we can still define geometry on them using inner products!

🔗

Subsection 5.4.2 Definition of Inner Product Space

From Dot Product to Inner Product:

🔗

Recall the four key properties of the dot product in \(\mathbb{R}^n\text{:}\)

\(\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}\) (Symmetry)

🔗
\(\mathbf{u} \cdot (\mathbf{v} + \mathbf{w}) = \mathbf{u} \cdot \mathbf{v} + \mathbf{u} \cdot \mathbf{w}\) (Linearity)

🔗
\(c(\mathbf{u} \cdot \mathbf{v}) = (c\mathbf{u}) \cdot \mathbf{v}\) (Homogeneity)

🔗
\(\mathbf{v} \cdot \mathbf{v} \geq 0\text{,}\) with equality iff \(\mathbf{v} = \mathbf{0}\) (Positive Definiteness)

🔗

🔗

We now use these as axioms to define inner products on any vector space:

🔗

Definition 5.4.1. Inner Product.

Let \(\mathbf{u}, \mathbf{v}\text{,}\) and \(\mathbf{w}\) be vectors in a vector space \(V\text{,}\) and let \(c\) be any scalar. An inner product on \(V\) is a function that associates a real number \(\langle\mathbf{u}, \mathbf{v}\rangle\) with each pair of vectors \(\mathbf{u}\) and \(\mathbf{v}\) and satisfies the following axioms:

\(\langle\mathbf{u}, \mathbf{v}\rangle=\langle\mathbf{v}, \mathbf{u}\rangle\) (Symmetry)
🔗

🔗
\(\langle\mathbf{u}, \mathbf{v}+\mathbf{w}\rangle=\langle\mathbf{u}, \mathbf{v}\rangle+\langle\mathbf{u}, \mathbf{w}\rangle\) (Additivity)
🔗

🔗
\(c\langle\mathbf{u}, \mathbf{v}\rangle=\langle c \mathbf{u}, \mathbf{v}\rangle\) (Homogeneity)
🔗

🔗
\(\langle\mathbf{v}, \mathbf{v}\rangle \geq 0\text{,}\) and \(\langle\mathbf{v}, \mathbf{v}\rangle=0\) if and only if \(\mathbf{v}=\mathbf{0}\) (Positive Definiteness)
🔗

🔗

🔗

Definition 5.4.2. Inner Product Space.

A vector space \(V\) with an inner product is called an inner product space.

🔗

Notation: We use angle brackets \(\langle\cdot, \cdot\rangle\) for a general inner product to distinguish it from the dot product \(\cdot\) in \(\mathbb{R}^n\text{.}\) Of course, in \(\mathbb{R}^n\text{,}\) we have \(\langle\mathbf{u}, \mathbf{v}\rangle = \mathbf{u} \cdot \mathbf{v}\text{.}\)

🔗

Example 5.4.3. Verifying an Inner Product.

Show that the function \(\langle\mathbf{u}, \mathbf{v}\rangle = u_1v_1 + 2u_2v_2\) defines an inner product on \(\mathbb{R}^2\text{,}\) where \(\mathbf{u} = (u_1, u_2)\) and \(\mathbf{v} = (v_1, v_2)\text{.}\)

🔗

Solution.

We verify each axiom:

🔗

Symmetry: \(\langle\mathbf{u}, \mathbf{v}\rangle = u_1v_1 + 2u_2v_2 = v_1u_1 + 2v_2u_2 = \langle\mathbf{v}, \mathbf{u}\rangle\) ✓
🔗

🔗
Additivity: Let \(\mathbf{w} = (w_1, w_2)\text{.}\) Then

\begin{align*} \langle\mathbf{u}, \mathbf{v}+\mathbf{w}\rangle \amp= u_1(v_1+w_1) + 2u_2(v_2+w_2)\\ \amp= u_1v_1 + u_1w_1 + 2u_2v_2 + 2u_2w_2\\ \amp= (u_1v_1 + 2u_2v_2) + (u_1w_1 + 2u_2w_2)\\ \amp= \langle\mathbf{u}, \mathbf{v}\rangle + \langle\mathbf{u}, \mathbf{w}\rangle \end{align*}

✓

🔗

🔗
Homogeneity: \(\langle c\mathbf{u}, \mathbf{v}\rangle = (cu_1)v_1 + 2(cu_2)v_2 = c(u_1v_1 + 2u_2v_2) = c\langle\mathbf{u}, \mathbf{v}\rangle\) ✓
🔗

🔗
Positive Definiteness: \(\langle\mathbf{v}, \mathbf{v}\rangle = v_1^2 + 2v_2^2 \geq 0\text{,}\) and equals 0 only if \(v_1 = v_2 = 0\text{,}\) i.e., \(\mathbf{v} = \mathbf{0}\text{.}\) ✓
🔗

🔗

🔗

All four axioms hold, so this defines an inner product on \(\mathbb{R}^2\text{.}\)

🔗

Example 5.4.4. A Function That Is Not an Inner Product.

Show that \(\langle\mathbf{u}, \mathbf{v}\rangle = u_1v_1 - 2u_2v_2 + u_3v_3\) is NOT an inner product on \(\mathbb{R}^3\text{.}\)

🔗

Solution.

We only need to find one axiom that fails. Let’s check positive definiteness with \(\mathbf{v} = (0, 1, 0)\text{:}\)

\begin{equation*} \langle\mathbf{v}, \mathbf{v}\rangle = 0^2 - 2(1)^2 + 0^2 = -2 \lt 0 \end{equation*}

🔗

This violates the positive definiteness axiom, so this is NOT an inner product.

🔗

Lemma 5.4.5. Standard Inner Products on Common Spaces.

On \(\mathcal{P}_n(x)\text{:}\) For polynomials \(p(x) = a_0+a_1x+\ldots+a_nx^n\) and \(q(x) = b_0+b_1x+\ldots+b_nx^n\text{,}\) the function

\begin{equation*} \langle p, q \rangle=a_0b_0+a_1b_1+\ldots+a_nb_n \end{equation*}

is an inner product.

🔗

Interpretation: This treats polynomials like vectors in \(\mathbb{R}^{n+1}\) by considering their coefficients.
🔗

🔗
On \(C[a,b]\text{:}\) For continuous functions \(f\) and \(g\text{,}\) the function

\begin{equation*} \langle f,g\rangle=\int_{a}^{b} f(x)g(x)\,dx \end{equation*}

is an inner product.

🔗

Interpretation: This measures how \"similar\" two functions are over the interval \([a,b]\text{.}\) Orthogonal functions have zero integral of their product\u2014they don’t \"overlap.\"
🔗

🔗
On \(M_{m,n}\) (matrix space): For matrices \(A = [a_{ij}]\) and \(B = [b_{ij}]\text{,}\) the function

\begin{equation*} \langle A, B \rangle = \sum_{i=1}^{m}\sum_{j=1}^{n} a_{ij}b_{ij} \end{equation*}

is an inner product.

🔗

Interpretation: This is the sum of products of corresponding entries\u2014like the dot product but for matrices.
🔗

🔗

🔗

Example 5.4.6. Orthogonal Functions: The Fourier Basis.

In \(C[0, 2\pi]\) with inner product \(\langle f, g \rangle = \int_0^{2\pi} f(x)g(x)\,dx\text{,}\) show that the set

\begin{equation*} S = \{1, \sin x, \cos x, \sin 2x, \cos 2x, \ldots, \sin nx, \cos nx\} \end{equation*}

is orthogonal.

🔗

Solution.

We need to show that \(\langle f, g \rangle = 0\) for any two distinct functions in \(S\text{.}\) For example:

🔗

\begin{equation*} \langle \sin x, \cos x \rangle = \int_0^{2\pi} \sin x \cos x\,dx = \frac{1}{2}\int_0^{2\pi} \sin 2x\,dx = 0 \end{equation*}

🔗

\begin{equation*} \langle \sin x, \sin 2x \rangle = \int_0^{2\pi} \sin x \sin 2x\,dx = 0 \end{equation*}

🔗

(using trigonometric identities and integration)

🔗

Why This Matters: This orthogonal set forms the basis for Fourier series, which decompose any periodic function into sine and cosine components\u2014fundamental to signal processing, solving differential equations, and analyzing vibrations!

🔗

Subsection 5.4.3 Gram-Schmidt Process in an Inner Product Space

Proposition 5.4.7.

\(\mathbf{P}_n(x)\) with the inner product in Lemma 5.4.5 is an inner product space.
🔗

🔗
\(C[a,b]\) with the inner product in Lemma 5.4.5 is an inner product space.
🔗

🔗

🔗

Theorem 5.4.8.

Let \(S=\{\mathbf{v}_1,\mathbf{v}_2,\ldots, \mathbf{v}_n\}\) be a basis of an inner product space \(V\text{.}\)

Let \(B^{\prime}=\left\{\mathbf{w}_{1}, \mathbf{w}_{2}, \ldots, \mathbf{w}_{n}\right\}\text{,}\) where

\begin{align*} w_1\amp=\mathbf{v}_{1} \\ w_2\amp=\mathbf{v}_{2}-\frac{\langle\mathbf{v}_{2}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}\\ w_3\amp=\mathbf{v}_{3}-\frac{\langle\mathbf{v}_{3}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}-\frac{\langle\mathbf{v}_{3}, \mathbf{w}_{2}\rangle}{\langle \mathbf{w}_{2}, \mathbf{w}_{2}\rangle} \mathbf{w}_{2} \\ \amp\vdots\\ w_m\amp=\mathbf{v}_{n}-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{1}\rangle}{\langle \mathbf{w}_{1}, \mathbf{w}_{1}\rangle} \mathbf{w}_{1}-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{2}\rangle}{\langle \mathbf{w}_{2}, \mathbf{w}_{2}\rangle} \mathbf{w}_{2}-\ldots-\frac{\langle\mathbf{v}_{n}, \mathbf{w}_{n-1}\rangle}{\langle \mathbf{w}_{n-1}, \mathbf{w}_{n-1}\rangle} \mathbf{w}_{n-1} \end{align*}

\(B'\) is an orthogonal basis of \(W\text{.}\)

🔗

🔗
Let \(\mathbf{u}_{i}=\frac{\mathbf{w}_{i}}{\left\|\mathbf{w}_{i}\right\|}\text{.}\) Then \(B^{\prime \prime}=\left\{\mathbf{u}_{1}, \mathbf{u}_{2}, \ldots, \mathbf{u}_{n}\right\}\) is an orthonormal basis for \(W\text{.}\) Also, for \(k=1,2, \ldots, m\text{,}\)

\begin{equation*} \operatorname{span}\left\{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots, \mathbf{v}_{k}\right\}=\operatorname{span}\left\{\mathbf{u}_{1}, \mathbf{u}_{2}, \ldots, \mathbf{u}_{k}\right\} \end{equation*}

🔗

🔗

🔗

Example: In \(C[0,2]\text{,}\) find an orthonormal basis of the space \(W=\operatorname{span}\{1,x,x^{2}\}\text{.}\)

🔗

Prev Top Next