High-dimensional data often varies along only a few directions. PCA finds the directions of greatest variance, so the data can be projected onto fewer dimensions without losing much.
Suppose you have to flatten 100-dimensional data down to 2 dimensions. If the data varies along only a handful of directions, the signal lives in the directions where it spreads out most. Principal component analysis (PCA) finds those directions: the axes of the cloud's shape, longest first, each perpendicular to the ones before.
For data in more than one dimension, variance depends on the direction you measure it in: project the points onto an axis and measure their spread, rotate the axis and the spread changes. Figure 1 shows 50 simulated, correlated points.
Figure 1. Simulated data. Drag the projection axis (or focus it and use the arrow keys) to rotate it through the cloud. Gray drop lines meet the axis at right angles, and the histogram shows the projected positions.
Lined up with the long direction of the ellipse, the histogram spreads widest; perpendicular to it, narrowest. The direction of maximum variance is the first principal component. Variance is what we keep because a direction where all the points overlap says nothing about how they differ.
PCA automates the dragging. It centers the data and computes the covariance matrix, which measures how much each pair of data variables moves together. Its eigenvectors point along the directions of greatest and least variance, and its eigenvalues give the variance along each.
Applied to a principal direction, the covariance matrix does not rotate it; it only scales it, by the variance along it.
Figure 2. PCA on the cloud from Figure 1, animated: find the mean, center the data, then grow PC1 along the direction of maximum variance and PC2 perpendicular to it. The bars show each eigenvalue's share of the total variance.
The result is a rotated coordinate system aligned with the data's own axes, in which the new coordinates are uncorrelated.
To reduce dimension, keep the first few components as the columns of a matrix W. The projection of a data point is a matrix multiply, and the reconstruction error is the squared distance between the point and its round trip back.
Figure 3 shows a 3D cloud shaped like a flattened ellipsoid. Drop PC3, its thinnest direction, and the cloud collapses to a plane at little cost; drop PC2 as well and it collapses to a line.
Figure 3. A simulated 3D cloud in isometric projection with its three principal axes. Toggle components off to project the data: faint gray dots mark where the points were, and red lines join each point to its projection.
The reconstruction error summed over all points equals the number of points times the variance of the dropped components: dropping only PC3 here costs 7.4, which is 100 points times PC3's variance of 0.074.
A scree plot shows each component's variance in order, with a running total, to help decide how many to keep. Components before the bend, the "elbow", tend to carry structure and those after it mostly noise, so the elbow is a common place to cut.
Figure 4. The principal components of 120 simulated points in 8 dimensions: bars for each share of the variance, a line for the running total. Below, points rebuilt from the kept components, on two of eight coordinates: gray originals, blue rebuilds, red errors. The elbow is at 2 to 3.
Elbows are not always sharp. Some practitioners instead keep enough components to explain 90% or 95% of the variance (3 and 5 components in Figure 4); others use cross-validation.
The elbow assumes that noise looks flat, and it does not. Take data in which every direction has variance exactly 1. With finitely many points, the eigenvalues of the sample covariance still spread out, over a range set by the ratio γ of dimensions to points: for γ = 1/2, from 0.09 to 2.91. This is the Marchenko-Pastur law (formula as in Benaych-Georges and Nadakuditi, 2011).
Figure 5. Simulated: the 200 eigenvalues of a sample covariance in 200 dimensions, every direction with variance 1 except one with 1 + θ. The curve is the Marchenko-Pastur density for pure noise. The purple marker is PC1's eigenvalue; the hollow one, its large-sample prediction.
One real direction with extra variance θ stands apart only if θ exceeds √γ. Below that, its eigenvalue sinks into the noise and a scree plot cannot tell it from noise; above it, the eigenvalue tends to (1 + θ)(1 + γ/θ) (Baik and Silverstein, as stated by Paul, 2007).
Figure 6. Simulated: the angle between PC1 and the true direction in 100 dimensions, five samples at each number of points n. The curve is Paul's large-sample limit, 90° up to the threshold n = p/θ2 (dashed). The dotted line marks a random direction's typical angle.
Finding the direction is harder than seeing the eigenvalue. Paul proved that below the same threshold, PC1 ends up perpendicular to the true direction; above it, the squared cosine of the angle tends to (1 − γ/θ2)/(1 + γ/θ). With θ = 0.5, a direction with 50% more variance than any other, 64 points per dimension still leave PC1 about 17° off.
PCA is not designed to find clusters or curved structure, though it separates clusters that differ along high-variance directions. A ring has equal variance in every direction, so PCA has no preferred axis, and any projection folds opposite sides of the ring together. The dimensionality reduction essay in the bioinformatics series runs PCA next to t-SNE on synthetic data.