Home / Courses / Linear Algebra / Unit 8

Linear Algebra Unit 8: Symmetric matrices & SVD

Symmetric matrices have especially nice structure. The singular value decomposition extends that structure to every matrix.

Covers: Spectral theorem, quadratic forms, singular value decomposition

Study guide

Symmetric matrices (A = Aᵀ) are especially well behaved. Their eigenvalues are all real, and eigenvectors from different eigenvalues are orthogonal. The spectral theorem says every symmetric matrix can be orthogonally diagonalized: A = QDQᵀ, with Q orthogonal (orthonormal eigenvector columns) and D diagonal. This writes A as a sum of simple projection pieces, each eigenvalue times a projection onto its eigenvector.

A quadratic form is an expression like Q(x) = xᵀAx with A symmetric, such as 3x² + 4xy + y². Changing variables to the eigenvector basis removes the cross terms, leaving λ₁y₁² + λ₂y₂² + …. The eigenvalues classify the form: all positive means positive definite (Q(x) > 0 for x ≠ 0), all negative means negative definite, and mixed signs means indefinite. The principal axes of an ellipse or hyperbola are the eigenvector directions.

The singular value decomposition generalizes diagonalization to every matrix, square or not: A = UΣVᵀ, where U and V are orthogonal and Σ holds the singular values on its diagonal. The singular values are the square roots of the eigenvalues of AᵀA, the columns of V are eigenvectors of AᵀA, and the columns of U are the normalized images Avᵢ divided by σᵢ. The number of nonzero singular values is the rank. Keeping only the largest singular values gives the best low-rank approximation of A, used in data compression and principal component analysis.

A symmetric matrix is positive definite exactly when all of its leading principal minors (the upper-left 1×1, 2×2, … determinants) are positive. Equivalently, all pivots in elimination are positive. A negative determinant, or any zero or negative diagonal entry, immediately rules it out. AᵀA (n×n) and AAᵀ (m×m) are both symmetric and positive semidefinite, and they have the same nonzero eigenvalues, σ₁², σ₂², …. The larger of the two just has extra zero eigenvalues.

In A = UΣVᵀ with rank r, the first r columns of V are an orthonormal basis for the row space and the remaining columns of V span the null space. The first r columns of U span the column space and the remaining columns of U span the left null space. When A is tall, finish U by adding an orthonormal basis for Nul Aᵀ.

Key ideas

• Spectral theorem: a symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors.
• So A = QDQᵀ with Q orthogonal.
• Quadratic forms xᵀAx are classified by the signs of A's eigenvalues.
• SVD: A = UΣVᵀ for any matrix; the singular values are square roots of AᵀA's eigenvalues.
• SVD powers data compression and principal component analysis.

Formulas and rules

Common mistakes

• Assuming non-symmetric matrices diagonalize orthogonally.
• Taking singular values from A's eigenvalues instead of AᵀA's.

Words to know

Spectral theorem

A symmetric matrix can be written A = QDQᵀ with Q orthogonal and D diagonal (real eigenvalues).

Singular value decomposition

A = UΣVᵀ

Singular values of A

Square roots of the eigenvalues of AᵀA.

Positive definite

A quadratic form (or symmetric matrix) with all eigenvalues positive.

Existence of the SVD

Every matrix, square or not, has a singular value decomposition.

Symmetric matrix

A = Aᵀ; real eigenvalues, orthogonal eigenvectors.

Orthogonally diagonalizable

A = QDQᵀ with Q orthogonal.

Quadratic form

Q(x) = xᵀAx with A symmetric.

Negative definite

All eigenvalues negative.

Indefinite

Eigenvalues of mixed signs.

Principal axes

The eigenvector directions that remove cross terms.

Low-rank approximation

Keeping only the largest singular values of A.

Sample practice questions

19 more questions like these, from level 1 to 5, are in the practice set.