PCA & SVD in Practice

The eigenvector article answered a theoretical question: which directions does a matrix leave alone? Practice asks a different one: given a cloud of data points, which directions carry the signal?

The recipe starts by centering: subtract the average point so the cloud sits around the origin. Then build the covariance matrix C = (1/n)·XᵀX. Here X is the table of centered points with one row per point, n is the number of points, and the small T means rows and columns are swapped. The covariance matrix records how each feature varies with every other. Its eigenvectors are the principal components, and its eigenvalues are the variance — the amount of spread — along each one.

The first principal component is the single direction that captures the most spread when the points are projected onto it. The second is the best direction at a right angle to the first, and so on. The cloud does not have to be two-dimensional: the covariance table grows with the number of features, but the ordering story stays the same.

The principal components are also the right singular vectors of the centered data matrix. The singular value decomposition writes any matrix as X = UΣVᵀ. V holds the principal directions. Σ (capital sigma) is a diagonal table of numbers called singular values — the square roots of the eigenvalues. U holds each point's coordinates in the new basis. SVD works on rectangular tables, not just square ones, which is why it is the version that ships in software.

The rank of a matrix is the number of genuinely independent directions it contains. For any rank r — keeping the top r directions and dropping the rest — X_r = U_rΣ_rV_rᵀ is the best possible rank-r approximation of the data, and the squared error equals exactly the sum of the squared singular values you dropped, Σ_{i>r} σ_i². That is the Eckart–Young theorem. Truncating an SVD is not a guess; it is provably optimal.

Preprocessing changes the answer, so it is part of the method. Centering moves the axes so they explain spread around the average point instead of around the origin. Standardizing makes each feature's spread equal to one before the analysis, which turns the covariance table into a correlation table. Then a feature measured in centimeters cannot dominate one measured in kilometers.

That is usually what you want when the units differ, and usually not when they do not. PCA on standardized data is a genuinely different decomposition: the axes swing and the eigenvalues change. The demo lets you watch it happen.

Start in the project tab and drag the tip of the direction arrow u. The dashed blue lines are the true principal axes for the current preprocessing choice, and the residual readout is the variance your chosen direction leaves on the table.

Drag u onto the first principal axis and the residual hits its lowest possible value, which equals the second eigenvalue. Every other direction explains less. That is what makes PCA a maximization problem rather than a change of basis you pick by eye.

Switch to reconstruct and keep one component. Every point collapses onto the first principal axis, and the warm segments show exactly what each point lost. Keep two and the rebuild is exact, because the space itself is two-dimensional. Toggle centering and standardization and watch the axes — and the reconstruction — change under the same cloud.

At scale nobody forms the full covariance table. Randomized SVD computes the top k components by multiplying the data by a small random sketch, cleaning up the result so its columns meet at right angles, and running a tiny decomposition on what is left. A few passes recover the dominant directions to high accuracy, and the cost grows with k instead of cubing with the dimension.

What people do with the components falls into four buckets. Denoise: drop the tail, where the noise lives. Compress: store only the top r coordinates. Whiten: rescale the components to equal variance so later models see uncorrelated features. Diagnose: read the scree plot, which shows how much variance each component holds, to see whether the data is closer to rank 3 or rank 300.

Low-rank thinking is everywhere in 2026. Matryoshka embeddings are trained so that cutting a 1024-dimension vector down to its first 256 coordinates keeps most of the retrieval quality — the PCA truncation argument baked into the loss. LoRA fine-tunes a model by learning a low-rank update ΔW = BA instead of the full matrix.

Neither trick runs an explicit SVD at inference time, but both are understandable only if you know what truncation costs. The pseudoinverse — the closest thing to an inverse for a table that cannot be inverted — is the same story from the other side: X⁺ = VΣ⁺Uᵀ, where tiny singular values are replaced rather than inverted — ridge regression in disguise.

The problems below center the data, build a rank-1 approximation, reconstruct at rank r, run power iteration for singular vectors, and compute the pseudoinverse and 2x2 singular values by hand.