Skip to main content
Home / Orthogonality / Gram Schmidt

Orthogonality

Orthogonal projections, bases, and distances

💡 Need Help?

  • • Enter fractions like 1/2
  • • Use decimals like 0.5
  • • Click "Load Example" to try a preset
  • • Click any step to copy LaTeX
Calculating...

Gram Schmidt Calculator

The Gram-Schmidt process converts a set of linearly independent vectors into an orthogonal (or orthonormal) basis that spans the same subspace. This is fundamental for QR decomposition, least squares, and many other linear algebra applications.

Vectors:
Dimension: D
Enter fractions like 1/2 or decimals like 0.5. The Gram-Schmidt process will orthogonalize these vectors step by step.

📑 Quick Navigation

What is the Gram-Schmidt Process?

The Gram-Schmidt process is a method for converting a set of linearly independent vectors into a set of orthogonal (or orthonormal) vectors that span the same subspace.

Core Idea: Starting with vectors {v₁, v₂, ..., vₖ}, the Gram-Schmidt process produces orthogonal vectors {u₁, u₂, ..., uₖ} where each uᵢ is orthogonal to all previous uⱼ (j < i).

$$ \boxed{\text{Given } \{v_1, v_2, ..., v_k\} \rightarrow \text{Gram-Schmidt} \rightarrow \{u_1, u_2, ..., u_k\} \text{ where } u_i \cdot u_j = 0 \text{ for } i \neq j} $$

💡 Memory Aid: "Gram-Schmidt takes a messy set of vectors and straightens them out into perpendicular directions."

Why Do We Need an Orthogonal Basis?

✅ Advantages of Orthogonal Bases

  • Easy coordinates: Finding coefficients is simple dot products
  • No linear dependencies: Each direction is independent
  • Numerical stability: Less rounding error in computations
  • QR decomposition: Foundation for solving least squares
  • Projections become simple: Just dot products!

❌ Problems with Non-Orthogonal Bases

  • Messy coordinates: Need to solve linear systems
  • Redundancy: Vectors may be nearly dependent
  • Numerical issues: Small errors become large
  • Hard to visualize: Skewed coordinate axes

Simple Example: In ℝ², the standard basis e₁ = (1,0) and e₂ = (0,1) is orthonormal. Any vector (x,y) has coordinates x and y directly. Try finding coordinates in a non-orthogonal basis like (1,0) and (1,1)—much harder!

The Algorithm Step by Step

Given linearly independent vectors v₁, v₂, ..., vₖ, we produce orthogonal vectors u₁, u₂, ..., uₖ:

Step 1: Start with the first vector

$$ u_1 = v_1 $$

Step 2: Remove projections onto previous vectors

$$ u_2 = v_2 - \frac{v_2 \cdot u_1}{u_1 \cdot u_1} u_1 $$

Step 3: Continue for each vector

$$ u_k = v_k - \sum_{j=1}^{k-1} \frac{v_k \cdot u_j}{u_j \cdot u_j} u_j $$

For Orthonormal Vectors: To get orthonormal vectors (unit length), normalize each uᵢ:

$$ e_i = \frac{u_i}{\|u_i\|} $$

💡 Geometric Interpretation: At each step, you subtract the projections of the current vector onto all previously computed orthogonal vectors. What remains is perpendicular to all of them!

Complete Example

Problem: Apply Gram-Schmidt to vectors in ℝ³:

$$ v_1 = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix},\quad v_2 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix},\quad v_3 = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} $$

Step 1: Set u₁ = v₁

$$ u_1 = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} $$

Step 2: Compute u₂

First, compute the projection of v₂ onto u₁:

$$ \text{proj}_{u_1}(v_2) = \frac{v_2 \cdot u_1}{u_1 \cdot u_1} u_1 = \frac{(1)(1) + (0)(1) + (1)(0)}{1^2 + 1^2 + 0^2} u_1 = \frac{1}{2} \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \\ 0.5 \\ 0 \end{bmatrix} $$
$$ u_2 = v_2 - \text{proj}_{u_1}(v_2) = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} - \begin{bmatrix} 0.5 \\ 0.5 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \\ -0.5 \\ 1 \end{bmatrix} $$

Step 3: Compute u₃

Compute projections onto u₁ and u₂:

$$ \text{proj}_{u_1}(v_3) = \frac{v_3 \cdot u_1}{u_1 \cdot u_1} u_1 = \frac{0+1+0}{2} u_1 = 0.5 \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.5 \\ 0.5 \\ 0 \end{bmatrix} $$
$$ \text{proj}_{u_2}(v_3) = \frac{v_3 \cdot u_2}{u_2 \cdot u_2} u_2 = \frac{0(0.5) + 1(-0.5) + 1(1)}{0.5^2 + (-0.5)^2 + 1^2} u_2 = \frac{0.5}{1.5} u_2 = \frac{1}{3} \begin{bmatrix} 0.5 \\ -0.5 \\ 1 \end{bmatrix} $$
$$ u_3 = v_3 - \text{proj}_{u_1}(v_3) - \text{proj}_{u_2}(v_3) = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} - \begin{bmatrix} 0.5 \\ 0.5 \\ 0 \end{bmatrix} - \begin{bmatrix} \frac{1}{6} \\ -\frac{1}{6} \\ \frac{1}{3} \end{bmatrix} = \begin{bmatrix} -\frac{2}{3} \\ \frac{2}{3} \\ \frac{2}{3} \end{bmatrix} $$

Step 4: Normalize (for orthonormal basis)

$$ e_1 = \frac{u_1}{\|u_1\|} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix},\quad e_2 = \frac{u_2}{\|u_2\|} = \frac{1}{\sqrt{1.5}} \begin{bmatrix} 0.5 \\ -0.5 \\ 1 \end{bmatrix},\quad e_3 = \frac{u_3}{\|u_3\|} = \frac{\sqrt{3}}{2} \begin{bmatrix} -\frac{2}{3} \\ \frac{2}{3} \\ \frac{2}{3} \end{bmatrix} $$

✅ Verification: Check that e₁·e₂ = 0, e₁·e₃ = 0, and e₂·e₃ = 0. All dot products should be zero!

Modified vs Classical Gram-Schmidt

Classical Gram-Schmidt

$$ u_k = v_k - \sum_{j=1}^{k-1} \frac{v_k \cdot u_j}{u_j \cdot u_j} u_j $$
  • Computes all projections using original vₖ
  • More prone to rounding errors
  • Loss of orthogonality in floating-point

Modified Gram-Schmidt

$$ u_k^{(1)} = v_k - \frac{v_k \cdot u_1}{u_1 \cdot u_1} u_1 $$ $$ u_k^{(2)} = u_k^{(1)} - \frac{u_k^{(1)} \cdot u_2}{u_2 \cdot u_2} u_2 $$
  • Subtracts projections sequentially
  • More numerically stable
  • Better orthogonality in practice

⚠️ Numerical Note: For ill-conditioned vectors (nearly dependent), Modified Gram-Schmidt is significantly more accurate. Our calculator uses Modified Gram-Schmidt by default!

Key Properties

Span Preservation

The orthogonal set {u₁, u₂, ..., uₖ} spans the same subspace as the original vectors {v₁, v₂, ..., vₖ}.

$$ \text{span}\{u_1, ..., u_k\} = \text{span}\{v_1, ..., v_k\} $$

Pythagorean Theorem

For any vector, the squared norm equals sum of squared projection lengths:

$$ \|v_k\|^2 = \sum_{j=1}^{k-1} \left(\frac{v_k \cdot u_j}{\|u_j\|}\right)^2 + \|u_k\|^2 $$

QR Decomposition

Gram-Schmidt is the foundation of QR decomposition:

$$ A = QR $$

where Q has orthonormal columns, R is upper triangular.

Real-World Applications

📊 Data Science & ML

  • QR Decomposition: Solving least squares problems
  • Principal Component Analysis (PCA): Finding orthogonal components
  • Feature Engineering: Creating uncorrelated features
  • Signal Processing: Separating mixed signals

🔬 Engineering & Physics

  • Quantum Mechanics: Orthogonal quantum states
  • Vibration Analysis: Normal modes are orthogonal
  • Control Theory: Controllability and observability
  • Structural Analysis: Orthogonal deformation modes

📐 Mathematics

  • Orthogonal Polynomials: Legendre, Chebyshev, Hermite
  • Functional Analysis: Orthogonal functions (Fourier series)
  • Numerical Analysis: Stable basis construction
  • Linear Algebra: Basis orthogonalization

🤖 Computer Graphics

  • Coordinate Frames: Building orthonormal bases
  • Camera Orientation: Orthogonal view axes
  • 3D Rotations: Orthogonal rotation matrices
  • Surface Normals: Computing perpendicular vectors

Special Cases & Limitations

Scenario What Happens What to Do
Vectors are linearly dependent Some uᵢ becomes zero Detect dependence; cannot orthogonalize full set
Vectors nearly dependent Numerical instability, loss of orthogonality Use Modified Gram-Schmidt or reorthogonalization
Large number of vectors Accumulated rounding errors Use Householder QR for better stability
Complex vectors Works the same with conjugate transpose Use Hermitian inner product

⚠️ Important: Gram-Schmidt requires linearly independent input vectors. If vectors are dependent, the process will produce a zero vector, indicating dependence.

Frequently Asked Questions

Q: Does Gram-Schmidt change the span of the vectors?

A: No! The orthogonal set spans exactly the same subspace as the original vectors. This is a key property—you lose no information.

Q: What's the difference between orthogonal and orthonormal?

A:

  • Orthogonal: Vectors are perpendicular (dot product = 0)
  • Orthonormal: Orthogonal + each vector has length 1
Use the "Normalize" option in our calculator to get orthonormal vectors!

Q: Can Gram-Schmidt fail?

A: It fails (produces zero vectors) when the input vectors are linearly dependent. The process detects dependence naturally—if any uᵢ = 0, the original vectors were dependent.

Q: Why do we need orthogonal bases?

A: Orthogonal bases make computations much easier! In an orthonormal basis, the coordinates of any vector are simply dot products with the basis vectors. No linear system solving needed!

Q: What's the connection to QR decomposition?

A: Gram-Schmidt directly gives QR decomposition: Q contains the orthonormal vectors, and R contains the projection coefficients. This is foundational for solving least squares problems.

Q: Is Gram-Schmidt the most stable method?

A: Not for large or ill-conditioned problems. Householder transformations and Givens rotations are more stable, but Gram-Schmidt is simpler and sufficient for many applications.

Practice Problems

Beginner

  1. Apply Gram-Schmidt to v₁ = (2,0), v₂ = (1,3) in ℝ².
  2. Are u₁ = (1,1) and u₂ = (1,-1) orthogonal? Orthonormal?
  3. Normalize u = (3,4) to get a unit vector.

Intermediate

  1. Apply Gram-Schmidt to v₁ = (1,2,2), v₂ = (-1,0,2), v₃ = (0,0,1).
  2. Check if v₁ = (1,1,1), v₂ = (1,0,1), v₃ = (0,1,0) are linearly independent using Gram-Schmidt.
  3. Find the QR decomposition of A = [v₁, v₂] where v₁ = (3,1), v₂ = (2,2).

Advanced

  1. Prove that Gram-Schmidt preserves the span of the original vectors.
  2. Why does Modified Gram-Schmidt have better numerical stability?
  3. Show that if vectors are linearly dependent, Gram-Schmidt produces a zero vector.
Click to reveal solutions

1. u₁ = (2,0), u₂ = (1,3) - (2,0)·(1,3)/(4) · (2,0) = (1,3) - 0.5·(2,0) = (0,3)

2. Dot product = 0 → orthogonal. Lengths = √2, not 1 → not orthonormal.

3. ê = (3/5, 4/5)

4. u₁ = (1,2,2), u₂ = (-4/3, -2/3, 2/3), u₃ = (1/2, 0, -1/2)

5. Independent (no zero vectors produced)

6. Q = [u₁/‖u₁‖, u₂/‖u₂‖], R = [‖u₁‖, projection coefficients]

7. Each uₖ is a linear combination of original v₁...vₖ, so span preserved.

8. Modified version removes projections earlier, reducing rounding error accumulation.

9. If dependent, some vₖ is combination of earlier vectors, making uₖ zero.

Summary

🎯 Key Takeaways

  • Purpose: Convert any set of independent vectors into orthogonal/orthonormal vectors
  • Formula: uₖ = vₖ - Σ (projection of vₖ onto previous uⱼ)
  • Span preserved: Orthogonal set spans same subspace
  • Normalization: Divide by length to get orthonormal
  • Modified version: More numerically stable for real computations
  • Applications: QR decomposition, least squares, PCA, quantum mechanics

💡 Quick Check: After Gram-Schmidt, always verify orthogonality: dot product between different uᵢ should be zero (or very close in floating-point)!

Try It Yourself!

Use the calculator above to orthogonalize your own vectors:

  1. Enter your vectors as rows or components
  2. Choose normalization (orthogonal vs orthonormal)
  3. Select algorithm (Classical vs Modified Gram-Schmidt)
  4. Click "Calculate" to see:
    • Each orthogonalization step
    • Projection calculations
    • Final orthogonal/orthonormal basis
    • Verification of orthogonality

Test these examples:

  • ℝ² example: v₁ = (2,0), v₂ = (1,3)
  • ℝ³ example: v₁ = (1,1,0), v₂ = (1,0,1), v₃ = (0,1,1)
  • Dependent test: v₁ = (1,2), v₂ = (2,4) (see what happens!)
  • Normalization test: Compare orthogonal vs orthonormal output

📐 Pro Tip: Use the "Load Example" dropdown to quickly try common cases. Watch how each step subtracts projections to create perpendicular vectors!

Related Orthogonality Operations

Gram-Schmidt
Orthogonalize vectors
Orthonormal Basis
Unit orthogonal basis
Decomposition
Parallel/perpendicular components
Projection
Project onto subspace