Singular Value Decomposition (SVD): Complete Guide with Applications

Quick Navigation

• What is Singular Value Decomposition?
• The SVD Formula: A = UΣVᵀ
• Geometric Interpretation
• How to Compute SVD
• Properties of SVD
• Real-World Applications
• Step-by-Step Examples
• Frequently Asked Questions
• Practice Problems
• Related Topics

---

What is Singular Value Decomposition?

Definition

Singular Value Decomposition (SVD) is a powerful matrix factorization that works for any matrix (square or rectangular, invertible or singular). It states that any real matrix $A$ can be factored as:

$$ \boxed{A = U \Sigma V^T} $$

Where:

• $U$ ($m \times m$) - Left singular vectors (orthogonal matrix, $U^T U = I$)
• $\Sigma$ ($m \times n$) - Diagonal matrix of singular values ($\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$)
• $V^T$ ($n \times n$) - Right singular vectors (orthogonal matrix, $V^T V = I$)

Why is SVD So Important?

Unlike eigenvalue decomposition (which only works for square matrices), SVD works for ANY matrix:

• Rectangular matrices ($m \times n$)
• Square matrices ($n \times n$)
• Rank-deficient matrices
• Singular matrices

This makes SVD the most general and numerically stable matrix decomposition in linear algebra.

Visual Representation

For a $2 \times 2$ matrix:

$$ A = \underbrace{\left[\begin{array}{cc} u_{11} & u_{12} \\ u_{21} & u_{22} \end{array}\right]}_{U} \underbrace{\left[\begin{array}{cc} \sigma_1 & 0 \\ 0 & \sigma_2 \end{array}\right]}_{\Sigma} \underbrace{\left[\begin{array}{cc} v_{11} & v_{21} \\ v_{12} & v_{22} \end{array}\right]}_{V^T} $$

For a $2 \times 3$ matrix (rectangular):

$$ A_{2\times 3} = \underbrace{U_{2\times 2}}_{2\times 2} \cdot \underbrace{\Sigma_{2\times 3}}_{2\times 3} \cdot \underbrace{V^T_{3\times 3}}_{3\times 3} $$

---

The SVD Formula: A = UΣVᵀ

Component 1: U - Left Singular Vectors

$U$ is an $m \times m$ orthogonal matrix:

• Columns are orthonormal vectors ($u_1, u_2, \dots, u_m$)
• $u_i \cdot u_j = 0$ when $i \neq j$
• $\|u_i\| = 1$ for all $i$
• $U^T U = I$

What they represent: The left singular vectors are the eigenvectors of $AA^T$. They represent the directions in the output space.

Component 2: Σ - Singular Values Matrix

$\Sigma$ is an $m \times n$ diagonal matrix:

• Singular values $\sigma_1, \sigma_2, \dots, \sigma_r$ on the main diagonal
• Sorted in descending order: $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$
• All other entries are zero

What they represent: The singular values measure how much the matrix "stretches" space along each principal direction. They are the square roots of the eigenvalues of $A^T A$.

Component 3: Vᵀ - Right Singular Vectors

$V^T$ is an $n \times n$ orthogonal matrix:

• Rows are orthonormal vectors ($v_1^T, v_2^T, \dots, v_n^T$)
• $V$ is orthogonal: $V V^T = I$
• $V^T = V^{-1}$

What they represent: The right singular vectors are the eigenvectors of $A^T A$. They represent the directions in the input space.

Dimension Summary

Matrix	Dimensions	Description
$A$	$m \times n$	Original matrix
$U$	$m \times m$	Left singular vectors
$\Sigma$	$m \times n$	Singular values (diagonal)
$V^T$	$n \times n$	Right singular vectors (transposed)
$r$	rank($A$)	Number of non-zero singular values

---

Geometric Interpretation

The Circle-to-Ellipse Transformation

For a $2 \times 2$ matrix, SVD reveals how a linear transformation works geometrically:

$$ \text{Circle} \xrightarrow{V^T} \text{Rotated Circle} \xrightarrow{\Sigma} \text{Ellipse} \xrightarrow{U} \text{Final Ellipse} $$

Step-by-step:

1. $V^T$ rotates the circle (changes orientation in the domain)
2. $\Sigma$ stretches the circle into an ellipse ($\sigma_1$ and $\sigma_2$ are the semi-axis lengths)
3. $U$ rotates the ellipse to its final orientation in the codomain

Key Insight

The singular values $\sigma_1$ and $\sigma_2$ tell you:

• $\sigma_1$ = length of the major axis of the ellipse
• $\sigma_2$ = length of the minor axis of the ellipse
• $\sigma_1 / \sigma_2$ = condition number (how "squished" the ellipse is)

If $\sigma_2 = 0$, the ellipse collapses to a line (matrix is singular/rank-deficient).

---

How to Compute SVD

Method 1: Via $A^T A$ and $AA^T$ (Conceptual)

Step 1: Compute $A^T A$ ($n \times n$) and $AA^T$ ($m \times m$)

Step 2: Find eigenvalues of $A^T A$

• Eigenvalues: $\lambda_1, \lambda_2, \dots, \lambda_n$
• Sort in descending order: $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_r > 0 = \lambda_{r+1} = \dots = \lambda_n$

Step 3: Compute singular values $$ \sigma_i = \sqrt{\lambda_i} \quad \text{for } i = 1, 2, \dots, r $$

Step 4: Find right singular vectors $V$

• $V$ contains the normalized eigenvectors of $A^T A$
• Arrange columns in the same order as eigenvalues

Step 5: Compute left singular vectors $U$

• For $\sigma_i > 0$: $u_i = \frac{1}{\sigma_i} A v_i$
• For zero singular values, extend to an orthonormal basis using Gram-Schmidt if necessary.

Method 2: Numerical Computation (What the Calculator Does)

For matrices up to $3 \times 3$, we can compute exactly. For larger matrices, numerical algorithms (like Golub-Kahan) are used for stability. The calculator handles this automatically.

Example Calculation (2×2 Matrix)

Matrix: $$ A = \left[\begin{array}{cc} 3 & 1 \\ 1 & 3 \end{array}\right] $$

Step 1: Compute $A^T A$ $$ A^T A = \left[\begin{array}{cc} 3 & 1 \\ 1 & 3 \end{array}\right] \left[\begin{array}{cc} 3 & 1 \\ 1 & 3 \end{array}\right] = \left[\begin{array}{cc} 10 & 6 \\ 6 & 10 \end{array}\right] $$

Step 2: Find eigenvalues Characteristic equation: $(10-\lambda)^2 - 36 = 0 \Rightarrow \lambda_1 = 16, \lambda_2 = 4$

Step 3: Compute singular values $$ \sigma_1 = \sqrt{16} = 4,\quad \sigma_2 = \sqrt{4} = 2 $$

Step 4: Find $V$ (eigenvectors of $A^T A$) For $\lambda_1=16$: $v_1 = \left[\begin{array}{c} 1/\sqrt{2} \\ 1/\sqrt{2} \end{array}\right]$ For $\lambda_2=4$: $v_2 = \left[\begin{array}{c} 1/\sqrt{2} \\ -1/\sqrt{2} \end{array}\right]$

$$ V = \left[\begin{array}{cc} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{array}\right] \implies V^T = \left[\begin{array}{cc} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{array}\right] $$

Step 5: Compute $U$ $$ u_1 = \frac{1}{4} A v_1 = \left[\begin{array}{c} 1/\sqrt{2} \\ 1/\sqrt{2} \end{array}\right], \quad u_2 = \frac{1}{2} A v_2 = \left[\begin{array}{c} 1/\sqrt{2} \\ -1/\sqrt{2} \end{array}\right] $$

Final SVD: $$ A = \left[\begin{array}{cc} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{array}\right] \left[\begin{array}{cc} 4 & 0 \\ 0 & 2 \end{array}\right] \left[\begin{array}{cc} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{array}\right]^T $$

---

Properties of SVD

1. Rank Revealing

The number of non-zero singular values equals the rank of the matrix:

• $\text{rank}(A) = r$ where $\sigma_r > 0$ and $\sigma_{r+1} = 0$

2. Condition Number

The ratio $\sigma_1 / \sigma_r$ measures how "ill-conditioned" a matrix is:

• $\kappa(A) = \sigma_1 / \sigma_r$
• Large $\kappa$ = ill-conditioned (sensitive to errors)
• $\kappa \approx 1$ = well-conditioned

3. Orthogonality

$U$ and $V$ are orthogonal matrices:

• $U^T U = I_m$ ($U$ preserves lengths)
• $V^T V = I_n$ ($V$ preserves lengths)

4. Matrix Norms

• Spectral norm: $\|A\|_2 = \sigma_1$ (largest singular value)
• Frobenius norm: $\|A\|_{F} = \sqrt{\sigma_1^2 + \sigma_2^2 + \dots + \sigma_r^2}$

5. Low-Rank Approximation

The best rank-$k$ approximation to $A$ (in terms of least squares error) is: $$ A_k = \sum_{i=1}^k \sigma_i u_i v_i^T $$ This is the foundation of PCA and data compression.

6. Pseudoinverse

The Moore-Penrose pseudoinverse is: $$ A^+ = V \Sigma^+ U^T $$ where $\Sigma^+$ has $1/\sigma_i$ on the diagonal for non-zero $\sigma_i$.

---

Real-World Applications

📸 Image Compression

SVD allows massive image compression by keeping only the largest singular values:

Original image: $1000 \times 1000$ pixels (1,000,000 numbers) Compressed: Keep only top 50 singular values and corresponding vectors Storage: $\approx 100,000$ numbers (90% compression!)

How it works:

• Each singular value represents the "importance" of a feature.
• Smaller singular values add fine details/noise.
• Drop small $\sigma$'s and reconstruct $\rightarrow$ slightly blurry but recognizable image.

🎯 Principal Component Analysis (PCA)

PCA uses SVD to find directions of maximum variance in high-dimensional data:

Step 1: Center the data (subtract mean) Step 2: Compute SVD of the data matrix Step 3: Principal components = columns of $V$ Step 4: Variance explained by $\sigma_i^2 / \sum \sigma_j^2$

Applications:

• Dimensionality reduction
• Data visualization (2D/3D projections)
• Noise reduction
• Feature extraction

👍 Recommendation Systems (Netflix Prize)

SVD powered the winning solution for the Netflix Prize:

Matrix: Users $\times$ Movies (very sparse) Decompose: $A \approx U \Sigma V^T$ Predict: Unknown ratings estimated by $u_i \cdot \Sigma \cdot v_j^T$

Why it works: SVD finds latent factors (genres, actors, directors) that explain user preferences.

🔬 Signal Processing

SVD separates signal from noise:

• Signal $\rightarrow$ Large singular values
• Noise $\rightarrow$ Small singular values
• Denoising: Keep only large $\sigma$'s and reconstruct.

📊 Statistics: Principal Component Regression

Combine PCA with regression:

1. Compute SVD of $X$ (feature matrix)
2. Keep top $k$ principal components
3. Regress on these components
4. Reduces overfitting and multicollinearity

🤖 Natural Language Processing (LSA)

Latent Semantic Analysis uses SVD to find topics in documents:

Term-Document matrix (terms $\times$ documents) $\rightarrow$ SVD Dimensions: Terms, Hidden Concepts, Documents Concepts: Automatically discovered topics

---

Step-by-Step Examples

Example 1: 2×2 Matrix (Rank 2)

Matrix: $$ A = \left[\begin{array}{cc} 4 & 0 \\ 0 & 2 \end{array}\right] $$

Exact SVD (already diagonal): $$ U = \left[\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right],\quad \Sigma = \left[\begin{array}{cc} 4 & 0 \\ 0 & 2 \end{array}\right],\quad V^T = \left[\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right] $$

Interpretation: Matrix stretches x-direction by 4, y-direction by 2. No rotation needed.

Example 2: 2×2 Matrix (Rank 1 - Singular)

Matrix: $$ A = \left[\begin{array}{cc} 2 & 4 \\ 1 & 2 \end{array}\right] $$

Observation: Second row = $0.5 \times$ first row, so rank = 1.

Singular values: $\sigma_1 = \sqrt{20} \approx 4.472$, $\sigma_2 = 0$

Interpretation: The ellipse collapses to a line (matrix is singular).

Example 3: 2×3 Rectangular Matrix

Matrix: $$ A = \left[\begin{array}{ccc} 1 & 2 & 3 \\ 4 & 5 & 6 \end{array}\right] $$

SVD dimensions:

• $U$: $2 \times 2$
• $\Sigma$: $2 \times 3$ (only $\sigma_1, \sigma_2$ non-zero)
• $V^T$: $3 \times 3$

Interpretation: Maps $\mathbb{R}^3 \rightarrow \mathbb{R}^2$, losing one dimension of information.

Example 4: 3×2 Tall Matrix

Matrix: $$ A = \left[\begin{array}{cc} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{array}\right] $$

SVD dimensions:

• $U$: $3 \times 3$
• $\Sigma$: $3 \times 2$ (only $\sigma_1, \sigma_2$ non-zero)
• $V^T$: $2 \times 2$

Interpretation: Maps $\mathbb{R}^2 \rightarrow \mathbb{R}^3$, embedding in higher-dimensional space.

Example 5: Numerical Instability (Ill-conditioned)

Matrix: $$ A = \left[\begin{array}{cc} 1 & 1 \\ 1 & 1.0001 \end{array}\right] $$

Condition number: $\kappa \approx 40,000$ (very ill-conditioned!) Meaning: Small changes in input cause huge changes in output.

---

Frequently Asked Questions

Q: What's the difference between SVD and eigenvalue decomposition?

A:

• Eigenvalue decomposition (EVD): Only for square matrices, $A = PDP^{-1}$.
• SVD: Works for ANY matrix (rectangular, singular, etc.), $A = U\Sigma V^T$.
• For symmetric positive definite matrices, SVD and EVD are identical ($U=V, \Sigma=D$).

Q: Are singular values always positive?

A: Yes! Singular values are defined to be non-negative ($\sigma_i \ge 0$). By convention, we list them in descending order.

Q: Why do we need both U and V?

A: $U$ and $V$ are different because $A$ can map between spaces of different dimensions:

• $V$ rotates the input space (domain)
• $\Sigma$ scales the axes
• $U$ rotates the output space (codomain)

Q: What does $\sigma = 0$ mean?

A: A zero singular value means the matrix is singular (not full rank). The number of zero $\sigma$'s equals the dimension of the null space.

Q: How many non-zero singular values can a matrix have?

A: At most $\min(m, n)$. The rank of $A$ equals the number of non-zero singular values.

Q: Is SVD unique?

A: Almost unique. Singular values are unique. $U$ and $V$ are unique up to sign changes (if singular values are distinct) or orthogonal transformations within equal singular values.

Q: Why is SVD more numerically stable than eigenvalue decomposition?

A: SVD avoids forming $A^T A$, which squares the condition number. Direct SVD algorithms (Golub-Kahan) are much more stable.

Q: Can SVD handle complex numbers?

A: Yes! The complex SVD replaces $U$ and $V$ with unitary matrices ($U^* U = I$) and $\Sigma$ remains real diagonal.

Q: What's the computational cost of SVD?

A:

• $2 \times 2$: $O(1)$ (exact formula)
• $3 \times 3$: $O(1)$ (exact)
• $n \times n$ for large $n$: $\sim O(n^3)$ (comparable to eigenvalue decomposition)

Q: How is SVD related to PCA?

A: PCA is essentially SVD on the mean-centered data matrix. The principal components are the columns of $V$, and variances are proportional to $\sigma_i^2$.

---

Practice Problems

Beginner Level

1. Compute SVD: Find $\Sigma$ for $$ A = \left[\begin{array}{cc} 3 & 0 \\ 0 & 5 \end{array}\right] $$

2. Identify dimensions: For a $4 \times 7$ matrix, what are the dimensions of $U$, $\Sigma$, and $V^T$?

3. Rank from SVD: If $\sigma = [5, 3, 0, 0]$, what's the rank?

Intermediate Level

4. Compute full SVD: Find $U, \Sigma, V^T$ for $$ A = \left[\begin{array}{cc} 0 & 2 \\ 0 & 0 \end{array}\right] $$

5. Rectangular SVD: $A$ is $3 \times 2$ with $\sigma_1 = 4, \sigma_2 = 1$. What are the dimensions of $U, \Sigma, V^T$?

6. Condition number: For $A$ with $\sigma_1 = 100, \sigma_2 = 1$, what's the condition number? What does this tell you?

Advanced Level

7. Low-rank approximation: For $A$ with SVD, find the best rank-1 approximation in terms of $A$'s entries.

8. SVD and pseudoinverse: Express $A^+$ (pseudoinverse) in terms of $U, \Sigma, V$.

9. Image compression: If an image requires 10,000 numbers to store, how many numbers are needed to store the SVD with $k=100$ singular values? (Assume square $100 \times 100$ for simplicity).

10. Prove: Show that $\|A\|_2 = \sigma_1$ (largest singular value).

Solutions

1. $$ \Sigma = \left[\begin{array}{cc} 5 & 0 \\ 0 & 3 \end{array}\right] $$ (Note: $\sigma_1 \ge \sigma_2$, so we swap 3 and 5)

2. $U$: $4 \times 4$, $\Sigma$: $4 \times 7$, $V^T$: $7 \times 7$

3. rank = 2 (two non-zero singular values)

4. $$ U = \left[\begin{array}{cc} 0 & 1 \\ 1 & 0 \end{array}\right],\quad \Sigma = \left[\begin{array}{cc} 2 & 0 \\ 0 & 0 \end{array}\right],\quad V^T = \left[\begin{array}{cc} 0 & 1 \\ 1 & 0 \end{array}\right] $$

5. $U$: $3 \times 3$, $\Sigma$: $3 \times 2$, $V^T$: $2 \times 2$

6. $\kappa = 100$ (ill-conditioned). Small errors in input are amplified 100×.

7. $A_1 = \sigma_1 u_1 v_1^T$

8. $A^+ = V \Sigma^+ U^T$ where $\Sigma^+$ has $1/\sigma_i$ on diagonal

9. $U$: $100 \times 100 = 10,000$, $\Sigma$: $100$ values, $V^T$: $100 \times 100 = 10,000$ $\rightarrow$ total $\approx 20,100$ numbers. (Note: This is actually more storage for full rank, but for compression we use $k \ll n$).

10. Proof uses definition of spectral norm: $\|A\|_2 = \max_{\|x\|=1} \|Ax\|$. From SVD, this maximum occurs at $x = v_1$ (first right singular vector), giving $\sigma_1$.

---

Summary

Key Takeaways

Concept	Explanation
SVD Formula	$A = U \Sigma V^T$ for ANY matrix
U	Left singular vectors (eigenvectors of $AA^T$)
Σ	Diagonal matrix of singular values $\sigma_1 \ge \sigma_2 \ge \dots$
Vᵀ	Right singular vectors (eigenvectors of $A^T A$)
Rank	Number of non-zero singular values
Condition Number	$\sigma_1 / \sigma_r$ (large = ill-conditioned)
Low-rank approx	Best rank-k approximation = $\sum_{i=1}^k \sigma_i u_i v_i^T$

When to Use SVD

• ✅ Data compression (images, video, audio)
• ✅ PCA and dimensionality reduction
• ✅ Recommendation systems
• ✅ Solving least squares problems
• ✅ Signal denoising
• ✅ Latent Semantic Analysis (text mining)
• ✅ Computing matrix rank reliably
• ✅ Finding pseudoinverse

When Alternative Methods Are Better

• ❌ Eigenvalues only (symmetric matrices) $\rightarrow$ use eigendecomposition
• ❌ Solving linear systems (small) $\rightarrow$ use Gaussian elimination
• ❌ Cholesky (positive definite) $\rightarrow$ faster
• ❌ QR decomposition (least squares) $\rightarrow$ sometimes faster

---

Related Topics

Explore these related concepts in our Linear Algebra Toolbox:

• Eigenvalue/Eigenvector Calculator - SVD reduces to eigendecomposition for symmetric matrices
• Matrix Diagonalization - For symmetric matrices, diagonalization = SVD
• Positive Definite Checker - Positive definite matrices have $\sigma_1 \ge \dots \ge \sigma_n > 0$
• Linear ODE System Solver - SVD for analyzing stability
• Least Squares Calculator - SVD solves least squares robustly

---

Try It Yourself!

Use the calculator above to explore SVD for your own matrices:

1. 2×2 matrices - See the circle-to-ellipse transformation
2. Rectangular matrices - Watch how dimensions change
3. Singular matrices - Observe zero singular values
4. Ill-conditioned matrices - See large condition numbers

Test these examples:

• $$ \left[\begin{array}{cc} 3 & 1 \\ 1 & 3 \end{array}\right] $$ - Well-conditioned, $\sigma_1=4, \sigma_2=2$
• $$ \left[\begin{array}{cc} 1 & 0 \\ 0 & 0.0001 \end{array}\right] $$ - Ill-conditioned, $\kappa=10,000$
• $$ \left[\begin{array}{ccc} 1 & 2 & 3 \\ 4 & 5 & 6 \end{array}\right] $$ - Rectangular, $2 \times 3$
• $$ \left[\begin{array}{cc} 1 & 1 \\ 1 & 1.0001 \end{array}\right] $$ - Nearly singular

Pro tip: The condition number $\kappa = \sigma_1 / \sigma_r$ tells you numerical stability:

• $\kappa < 100$: Well-conditioned 👍
• $100 < \kappa < 10,000$: Moderately conditioned ⚠️
• $\kappa > 10,000$: Ill-conditioned (dangerous) 🔴