Singular Value Decomposition (SVD)

Section 9.2 Singular Value Decomposition (SVD)

Let \(\mathbf{A}\) be an \(m \times n\) matrix and rank \(\mathbf{A} = r\text{.}\) So the number of non-zero singular values of \(\mathbf{A}\) is \(r\text{.}\) Since they are positive and labeled in decreasing order, we can write them as:

\begin{equation*} \sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0 \end{equation*}

where:

\begin{equation*} \sigma_{r+1} = \sigma_{r+2} = \cdots = \sigma_n = 0 \end{equation*}

🔗

Theorem 9.2.1. Singular Value Decomposition Theorem.

Now we can write the singular value decomposition of \(\mathbf{A}\) as:

\begin{equation*} \mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T \end{equation*}

where:

\(\mathbf{V}\) is an \(n \times n\) matrix that its columns are \(\mathbf{v}_i\)

🔗
\(\boldsymbol{\Sigma}\) is an \(m \times n\) diagonal matrix with singular values on the diagonal

🔗
\(\mathbf{U}\) is an \(m \times m\) orthogonal matrix

🔗

🔗

We call a set of orthogonal and normalized vectors an orthonormal set. So the set \(\{\mathbf{v}_i\}\) is an orthonormal set. A matrix whose columns are an orthonormal set is called an orthogonal matrix, and \(\mathbf{V}\) is an orthogonal matrix.

🔗

\(\boldsymbol{\Sigma}\) is an \(m \times n\) diagonal matrix of the form:

\begin{equation*} \boldsymbol{\Sigma} = \begin{bmatrix} \sigma_1 & 0 & \cdots & 0 & 0 & \cdots & 0\\ 0 & \sigma_2 & \cdots & 0 & 0 & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \sigma_r & 0 & \cdots & 0\\ 0 & 0 & \cdots & 0 & 0 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & 0 & 0 & \cdots & 0 \end{bmatrix} \end{equation*}

🔗

We also know that the set \(\{\mathbf{A}\mathbf{v}_1, \mathbf{A}\mathbf{v}_2, \ldots, \mathbf{A}\mathbf{v}_r\}\) is an orthogonal basis for Col \(\mathbf{A}\text{,}\) and \(\sigma_i = \|\mathbf{A}\mathbf{v}_i\|\text{.}\) So we can normalize the \(\mathbf{A}\mathbf{v}_i\) vectors by dividing them by their length:

\begin{equation*} \mathbf{u}_i = \frac{\mathbf{A}\mathbf{v}_i}{\|\mathbf{A}\mathbf{v}_i\|} = \frac{\mathbf{A}\mathbf{v}_i}{\sigma_i}, \quad 1 \leq i \leq r \end{equation*}

🔗

The SVD equation can be simplified to:

\begin{equation*} \mathbf{A} = \sigma_1\mathbf{u}_1\mathbf{v}_1^T + \sigma_2\mathbf{u}_2\mathbf{v}_2^T + \cdots + \sigma_r\mathbf{u}_r\mathbf{v}_r^T \end{equation*}

🔗

Example 9.2.2. SVD Computation Example.

🔗

Prev Top Next