Singular Values

Section 9.1 Singular Values

Before talking about SVD, we should find a way to calculate the stretching directions for a non-symmetric matrix. Suppose that \(\mathbf{A}\) is an \(m \times n\) matrix which is not necessarily symmetric. Then it can be shown that:

\begin{equation*} \mathbf{A}^T\mathbf{A} \end{equation*}

is an \(n \times n\) symmetric matrix.

🔗

Example 9.1.1. Eigendecomposition of a 2×2 Matrix.

Let \(\mathbf{A} = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}\text{.}\) We compute its eigendecomposition.

🔗

First compute the eigenvalues from

\begin{equation*} \det(\mathbf{A} - \lambda \mathbf{I}) = 0 \end{equation*}

i.e.

\begin{equation*} \begin{vmatrix} 4-\lambda & 1 \\ 2 & 3-\lambda \end{vmatrix} = (4-\lambda)(3-\lambda) - 2 = \lambda^2 - 7\lambda + 10 = 0 \end{equation*}

giving \(\lambda_1 = 5\) and \(\lambda_2 = 2\text{.}\)

🔗

For \(\lambda_1 = 5\text{,}\) solving \((\mathbf{A} - 5\mathbf{I})\mathbf{x} = 0\) yields eigenvector \(\mathbf{v}_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}\text{.}\) For \(\lambda_2 = 2\text{,}\) solving \((\mathbf{A} - 2\mathbf{I})\mathbf{x} = 0\) yields eigenvector \(\mathbf{v}_2 = \begin{bmatrix} -1 \\ 2 \end{bmatrix}\text{.}\)

🔗

Form \(\mathbf{P} = [\mathbf{v}_1\ \mathbf{v}_2] = \begin{bmatrix} 1 & -1 \\ 1 & 2 \end{bmatrix}\) and \(\mathbf{D} = \begin{bmatrix} 5 & 0 \\ 0 & 2 \end{bmatrix}\text{.}\) Then \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\text{.}\)

🔗

Verification:

\begin{equation*} \mathbf{P}\mathbf{D}\mathbf{P}^{-1} = \begin{bmatrix} 1 & -1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 5 & 0 \\ 0 & 2 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 2 \end{bmatrix}^{-1} = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix} \end{equation*}

🔗

Thus we have completed the eigendecomposition of \(\mathbf{A}\text{.}\)

🔗

Remember that the transpose of a product is the product of the transposes in the reverse order. So:

\begin{equation*} (\mathbf{A}^T\mathbf{A})^T = \mathbf{A}^T(\mathbf{A}^T)^T = \mathbf{A}^T\mathbf{A} \end{equation*}

So \(\mathbf{A}^T\mathbf{A}\) is equal to its transpose, and it is a symmetric matrix.

🔗

Let \(\mathbf{A}\) be an \(m \times n\) matrix. We showed that \(\mathbf{A}^T\mathbf{A}\) is a symmetric matrix, so it has \(n\) real eigenvalues and \(n\) linear independent and orthogonal eigenvectors which can form a basis for the \(n\)-element vectors that it can transform (in \(\mathbb{R}^n\) space). We call these eigenvectors \(\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n\) and we assume they are normalized.

🔗

For each of these eigenvectors we can use the definition of length and the rule for the product of transposed matrices to have:

\begin{equation*} \|\mathbf{A}\mathbf{v}_i\|^2 = (\mathbf{A}\mathbf{v}_i)^T(\mathbf{A}\mathbf{v}_i) = \mathbf{v}_i^T\mathbf{A}^T\mathbf{A}\mathbf{v}_i \end{equation*}

🔗

Now we assume that the corresponding eigenvalue of \(\mathbf{v}_i\) is \(\lambda_i\text{:}\)

\begin{equation*} \mathbf{v}_i^T\mathbf{A}^T\mathbf{A}\mathbf{v}_i = \mathbf{v}_i^T\lambda_i\mathbf{v}_i = \lambda_i\mathbf{v}_i^T\mathbf{v}_i \end{equation*}

🔗

But \(\mathbf{v}_i\) is normalized, so:

\begin{equation*} \|\mathbf{v}_i\|^2 = \mathbf{v}_i^T\mathbf{v}_i = 1 \end{equation*}

As a result:

\begin{equation*} \|\mathbf{A}\mathbf{v}_i\|^2 = \lambda_i\mathbf{v}_i^T\mathbf{v}_i = \lambda_i \end{equation*}

🔗

This result shows that all the eigenvalues are positive. Now assume that we label them in decreasing order, so:

\begin{equation*} \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n \geq 0 \end{equation*}

🔗

Definition 9.1.2. Singular Value.

We define the singular value of \(\mathbf{A}\) as the square root of \(\lambda_i\) (the eigenvalue of \(\mathbf{A}^T\mathbf{A}\)), and we denote it with \(\sigma_i\text{:}\)

\begin{equation*} \sigma_i = \sqrt{\lambda_i} = \|\mathbf{A}\mathbf{v}_i\| \end{equation*}

\begin{equation*} \sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_n \geq 0 \end{equation*}

🔗

Prev Top Next