Home / Courses / Linear Algebra / Unit 7

Linear Algebra Unit 7: Orthogonality & least squares

Orthogonality brings geometry into linear algebra: lengths, angles, projections and the best approximate solution to systems with no exact answer.

Covers: Inner products, orthogonal projections, Gram-Schmidt, least squares

Study guide

The inner product (dot product) u·v measures lengths and angles: |v| = √(v·v), and u·v = |u||v|cos θ. Vectors are orthogonal when u·v = 0. A set of mutually orthogonal nonzero vectors is automatically linearly independent, and an orthonormal set also has unit vectors.

The projection of y onto a vector u is (y·u/u·u)u, the closest point to y on the line through u. Projection onto a subspace W with an orthogonal basis adds the projections onto each basis vector. The difference y − proj_W y is orthogonal to W, so the projection is the best approximation to y in W.

The Gram–Schmidt process turns any basis into an orthogonal basis: keep the first vector, then subtract from each new vector its projections onto the vectors already built. Normalizing gives an orthonormal basis. Applying it to the columns of A gives the QR factorization A = QR.

An orthogonal matrix Q has orthonormal columns, so QᵀQ = I and Q⁻¹ = Qᵀ. Orthogonal matrices preserve lengths and angles (rotations and reflections).

Least squares finds the best approximate solution when Ax = b has no exact solution. The least-squares solution x̂ minimizes |b − Ax|, and it satisfies the normal equations AᵀAx̂ = Aᵀb. This is how lines and curves are fitted to data: Ax̂ is the projection of b onto the column space of A.

If A has linearly independent columns, the matrix that projects onto Col A is P = A(AᵀA)⁻¹Aᵀ. It satisfies P² = P and Pᵀ = P, and its eigenvalues are only 1 and 0. Onto a single line through a, it simplifies to P = aaᵀ/(aᵀa). The row space and null space of A are orthogonal complements in ℝⁿ, and the column space and left null space are orthogonal complements in ℝᵐ. Their dimensions add up to n and to m.

If Q has orthonormal columns, the normal equations simplify to x̂ = Qᵀb, and the projection of b onto Col Q is QQᵀb. If the columns are orthogonal but not unit length, then AᵀA is diagonal, so the normal equations are easy to solve.

Key ideas

• Vectors are orthogonal when their dot product is 0.
• Projection of y onto u keeps only the part of y in u's direction.
• Gram–Schmidt turns any basis into an orthogonal one.
• Orthogonal matrices have QᵀQ = I, so Q⁻¹ = Qᵀ.
• Least squares solves AᵀAx̂ = Aᵀb when Ax = b has no solution.

Formulas and rules

Common mistakes

• Forgetting to normalize when an orthonormal basis is asked for.
• Solving Ax = b directly when it's inconsistent instead of using the normal equations.

Words to know

Projection of y onto u

(y·u / u·u)·u

Normal equations for least squares

AᵀA x̂ = Aᵀb

Orthogonal matrix

A square matrix with orthonormal columns; Q⁻¹ = Qᵀ.

Gram–Schmidt process

Turns a basis into an orthogonal (or orthonormal) basis.

Orthogonal vectors

Dot product equals 0.

Inner product

u·v; gives lengths |v| = √(v·v) and angles.

Orthogonal set

Mutually orthogonal nonzero vectors; automatically independent.

Orthonormal set

An orthogonal set of unit vectors.

Orthogonal projection onto W

The closest point in W to y; y − proj_W y is orthogonal to W.

QR factorization

A = QR, from Gram–Schmidt on the columns of A.

Least-squares solution

x̂ minimizing |b − Ax|.

Orthogonal complement (W⊥)

All vectors orthogonal to every vector in W.

Sample practice questions

21 more questions like these, from level 1 to 5, are in the practice set.